[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:14:54 dist-test-slave-j6q7 krb5kdc[6990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:14:54 dist-test-slave-j6q7 krb5kdc[6990](info): set up 2 sockets
Sep 26 15:14:54 dist-test-slave-j6q7 krb5kdc[6990](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 15:14:55.439092 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.004s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:14:55 dist-test-slave-j6q7 krb5kdc[6990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:14:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:14:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:14:55.600226 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:40995
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:36415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:40995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:55.885289 7003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:55.885514 7003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:55.885666 7003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:55.891846 7003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:55.891924 7003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:55.892091 7003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:55.892180 7003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:55.892290 7003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:55.901021 7003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:36415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:40995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:40995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:55.902180 7003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:55.903373 7003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:55.914724 7008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:55.915475 7011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:55.915563 7009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:55.918109 7003 server_base.cc:1034] running on GCE node
I20260926 15:14:55.918888 7003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:55.920563 7003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:55.921749 7003 hybrid_clock.cc:648] HybridClock initialized: now 1790435695921696 us; error 42 us; skew 500 ppm
I20260926 15:14:55.922079 7003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:55 dist-test-slave-j6q7 krb5kdc[6990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435695, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:55.924876 7003 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:14:55.926277 7003 webserver.cc:460] Webserver started at http://127.6.209.254:40615/ using document root <none> and password file <none>
I20260926 15:14:55.926882 7003 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:55.926997 7003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:55.927271 7003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:55.929849 7003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "27fdb401cfcc4f93bc95151d8c2d5cdf"
format_stamp: "Formatted at 2026-09-26 15:14:55 on dist-test-slave-j6q7"
server_key: "9b14949aeee13dcd10ec1513c33104d6"
server_key_iv: "3bdfc99562653809dfa6f2beca4e8828"
server_key_version: "encryptionkey@0"
I20260926 15:14:55.930483 7003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "27fdb401cfcc4f93bc95151d8c2d5cdf"
format_stamp: "Formatted at 2026-09-26 15:14:55 on dist-test-slave-j6q7"
server_key: "9b14949aeee13dcd10ec1513c33104d6"
server_key_iv: "3bdfc99562653809dfa6f2beca4e8828"
server_key_version: "encryptionkey@0"
I20260926 15:14:55.934885 7003 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 15:14:55.937717 7018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:55.938809 7003 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:14:55.938990 7003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "27fdb401cfcc4f93bc95151d8c2d5cdf"
format_stamp: "Formatted at 2026-09-26 15:14:55 on dist-test-slave-j6q7"
server_key: "9b14949aeee13dcd10ec1513c33104d6"
server_key_iv: "3bdfc99562653809dfa6f2beca4e8828"
server_key_version: "encryptionkey@0"
I20260926 15:14:55.939154 7003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:55.953995 7003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:55.957193 7003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:55.957427 7003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:55.974126 7003 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:40995
I20260926 15:14:55.974150 7069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:40995 every 8 connection(s)
I20260926 15:14:55.975190 7003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:14:55.978168 7070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:55.983444 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 7003
I20260926 15:14:55.983745 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:14:55.985845 6983 external_mini_cluster.cc:1428] Setting key b13ebeb0c4cb17e73ac63f39e91b2efc
I20260926 15:14:55.988756 7070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27fdb401cfcc4f93bc95151d8c2d5cdf: Bootstrap starting.
I20260926 15:14:55.991555 7070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27fdb401cfcc4f93bc95151d8c2d5cdf: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:55.992584 7070 log.cc:824] T 00000000000000000000000000000000 P 27fdb401cfcc4f93bc95151d8c2d5cdf: Log is configured to *not* fsync() on all Append() calls
I20260926 15:14:55.995170 7070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27fdb401cfcc4f93bc95151d8c2d5cdf: No bootstrap required, opened a new log
I20260926 15:14:56.003525 7070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27fdb401cfcc4f93bc95151d8c2d5cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27fdb401cfcc4f93bc95151d8c2d5cdf" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40995 } }
I20260926 15:14:56.003837 7070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27fdb401cfcc4f93bc95151d8c2d5cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:14:56.003930 7070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27fdb401cfcc4f93bc95151d8c2d5cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27fdb401cfcc4f93bc95151d8c2d5cdf, State: Initialized, Role: FOLLOWER
I20260926 15:14:56.004487 7070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27fdb401cfcc4f93bc95151d8c2d5cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27fdb401cfcc4f93bc95151d8c2d5cdf" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40995 } }
I20260926 15:14:56.004686 7070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27fdb401cfcc4f93bc95151d8c2d5cdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:14:56.004814 7070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27fdb401cfcc4f93bc95151d8c2d5cdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:14:56.004992 7070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27fdb401cfcc4f93bc95151d8c2d5cdf [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:56.007648 7070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27fdb401cfcc4f93bc95151d8c2d5cdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27fdb401cfcc4f93bc95151d8c2d5cdf" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40995 } }
I20260926 15:14:56.008090 7070 leader_election.cc:302] T 00000000000000000000000000000000 P 27fdb401cfcc4f93bc95151d8c2d5cdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27fdb401cfcc4f93bc95151d8c2d5cdf; no voters:
I20260926 15:14:56.008665 7070 leader_election.cc:288] T 00000000000000000000000000000000 P 27fdb401cfcc4f93bc95151d8c2d5cdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:14:56.008800 7075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27fdb401cfcc4f93bc95151d8c2d5cdf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:14:56.010146 7075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27fdb401cfcc4f93bc95151d8c2d5cdf [term 1 LEADER]: Becoming Leader. State: Replica: 27fdb401cfcc4f93bc95151d8c2d5cdf, State: Running, Role: LEADER
I20260926 15:14:56.010673 7070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27fdb401cfcc4f93bc95151d8c2d5cdf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:14:56.010634 7075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27fdb401cfcc4f93bc95151d8c2d5cdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27fdb401cfcc4f93bc95151d8c2d5cdf" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40995 } }
I20260926 15:14:56.014325 7076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27fdb401cfcc4f93bc95151d8c2d5cdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27fdb401cfcc4f93bc95151d8c2d5cdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27fdb401cfcc4f93bc95151d8c2d5cdf" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40995 } } }
I20260926 15:14:56.014707 7076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27fdb401cfcc4f93bc95151d8c2d5cdf [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:56.015022 7077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27fdb401cfcc4f93bc95151d8c2d5cdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27fdb401cfcc4f93bc95151d8c2d5cdf. Latest consensus state: current_term: 1 leader_uuid: "27fdb401cfcc4f93bc95151d8c2d5cdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27fdb401cfcc4f93bc95151d8c2d5cdf" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40995 } } }
I20260926 15:14:56.015275 7077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27fdb401cfcc4f93bc95151d8c2d5cdf [sys.catalog]: This master's current role is: LEADER
Sep 26 15:14:56 dist-test-slave-j6q7 krb5kdc[6990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:14:56.015604 7084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:14:56.030270 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:55.990123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54654 (local address 127.6.209.254:40995)
0926 15:14:55.990872 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:55.990884 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:55.992003 (+ 1119us) server_negotiation.cc:407] Connection header received
0926 15:14:56.001338 (+ 9335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:56.001362 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:56.001778 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:56.002208 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:56.005229 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:56.007045 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:56.008669 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:56.009725 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:56.010468 (+ 743us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:56.017202 (+ 6734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:56.017214 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:56.017234 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:56.017283 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:56.022129 (+ 4846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:56.023357 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:56.023368 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:56.023378 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:56.023610 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:56.024209 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:56.024216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:56.024222 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:56.024922 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:56.027930 (+ 3008us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:56.028243 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 15:14:56.028714 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":345,"threads_started":1}
I20260926 15:14:56.031024 7084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:14:56.039080 7084 catalog_manager.cc:1347] Generated new cluster ID: 8629b579126640b08fbc17d003ca8485
I20260926 15:14:56.039167 7084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:14:56.045904 7084 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:14:56.046851 7084 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:14:56.051301 7084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27fdb401cfcc4f93bc95151d8c2d5cdf: Generated new TSK 0
I20260926 15:14:56.051872 7084 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
I20260926 15:14:56.087983 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:40995
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:36415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:56.378587 7096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:56.378800 7096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:56.378881 7096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:56.385535 7096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:56.385632 7096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:56.385788 7096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:14:56.393829 7096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:36415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:40995
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:56.394798 7096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:56.395867 7096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:56.407567 7101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:56.407612 7104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:56.407572 7102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:56.408421 7096 server_base.cc:1034] running on GCE node
I20260926 15:14:56.408963 7096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:56.409624 7096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:56.410800 7096 hybrid_clock.cc:648] HybridClock initialized: now 1790435696410751 us; error 41 us; skew 500 ppm
I20260926 15:14:56.411057 7096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:56 dist-test-slave-j6q7 krb5kdc[6990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435696, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:56.414149 7096 init.cc:377] Logged in from keytab as kudu/127.6.209.193@KRBTEST.COM (short username kudu)
I20260926 15:14:56.415336 7096 webserver.cc:460] Webserver started at http://127.6.209.193:46485/ using document root <none> and password file <none>
I20260926 15:14:56.415911 7096 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:56.416033 7096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:56.416282 7096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:56.418821 7096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "1d03d83677c84e329f758522002dd572"
format_stamp: "Formatted at 2026-09-26 15:14:56 on dist-test-slave-j6q7"
server_key: "8f29a9353c13449dd40ca8c19af970d6"
server_key_iv: "2373ea2f11b89d41ab11243e6b94e887"
server_key_version: "encryptionkey@0"
I20260926 15:14:56.419530 7096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "1d03d83677c84e329f758522002dd572"
format_stamp: "Formatted at 2026-09-26 15:14:56 on dist-test-slave-j6q7"
server_key: "8f29a9353c13449dd40ca8c19af970d6"
server_key_iv: "2373ea2f11b89d41ab11243e6b94e887"
server_key_version: "encryptionkey@0"
I20260926 15:14:56.423913 7096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 15:14:56.426641 7111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:56.427716 7096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 15:14:56.427863 7096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "1d03d83677c84e329f758522002dd572"
format_stamp: "Formatted at 2026-09-26 15:14:56 on dist-test-slave-j6q7"
server_key: "8f29a9353c13449dd40ca8c19af970d6"
server_key_iv: "2373ea2f11b89d41ab11243e6b94e887"
server_key_version: "encryptionkey@0"
I20260926 15:14:56.428023 7096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:56.439395 7096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:56.442735 7096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:56.443022 7096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:56.443851 7096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:14:56.444911 7096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:14:56.444990 7096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:56.445096 7096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:14:56.445159 7096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:56.473453 7096 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:44101
I20260926 15:14:56.473515 7223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:44101 every 8 connection(s)
I20260926 15:14:56.474439 7096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:14:56.479925 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 7096
I20260926 15:14:56.480116 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:14:56.480623 6983 external_mini_cluster.cc:1428] Setting key a503831f16396eb7fe2682ebb0d35afc
Sep 26 15:14:56 dist-test-slave-j6q7 krb5kdc[6990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435696, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:14:56.492985 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:56.477088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:57499 (local address 127.6.209.254:40995)
0926 15:14:56.477239 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:56.477244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:56.478196 (+ 952us) server_negotiation.cc:407] Connection header received
0926 15:14:56.479646 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:56.479651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:56.479726 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:56.479818 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:56.481148 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:56.481957 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:56.483610 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:56.484356 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:56.484520 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:56.487702 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:56.487706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:56.487709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:56.487734 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:56.490123 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:56.490889 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:56.490892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:56.490894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:56.490993 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:56.491427 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:56.491430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:56.491432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:56.491664 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:56.491756 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:56.492551 (+ 795us) server_negotiation.cc:299] Negotiation successful
0926 15:14:56.492704 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:14:56.494633 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:56.477498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40995 (local address 127.6.209.193:57499)
0926 15:14:56.478069 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:56.478100 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 15:14:56.479415 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:14:56.480030 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:14:56.480041 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:14:56.480488 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:14:56.481010 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:56.481025 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:56.482195 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:56.482201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:56.483466 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:56.483478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:56.484546 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:56.484567 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:56.484755 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:56.485392 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:14:56.485410 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:14:56.487494 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:14:56.490243 (+ 2749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:56.490253 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:56.490267 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:56.490789 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:56.491086 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:56.491092 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:56.491098 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:56.491320 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:56.491773 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:14:56.491782 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:14:56.492176 (+ 394us) client_negotiation.cc:769] Sending connection context
0926 15:14:56.492477 (+ 301us) client_negotiation.cc:240] Negotiation successful
0926 15:14:56.493048 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":211,"threads_started":1}
WARNING: no policy specified for kudu/127.6.209.194@KRBTEST.COM; defaulting to no policy
I20260926 15:14:56.495983 7224 heartbeater.cc:347] Connected to a master server at 127.6.209.254:40995
I20260926 15:14:56.496212 7224 heartbeater.cc:464] Registering TS with master...
I20260926 15:14:56.496836 7224 heartbeater.cc:511] Master 127.6.209.254:40995 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.194@KRBTEST.COM" created.
I20260926 15:14:56.498626 7035 ts_manager.cc:194] Registered new tserver with Master: 1d03d83677c84e329f758522002dd572 (127.6.209.193:44101)
I20260926 15:14:56.499914 7035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.193@KRBTEST.COM'} at 127.6.209.193:57499
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
I20260926 15:14:56.515831 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:40995
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:36415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:56.813150 7229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:56.813395 7229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:56.813541 7229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:56.820075 7229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:56.820145 7229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:56.820263 7229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:14:56.828295 7229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:36415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:40995
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:56.829306 7229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:56.830348 7229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:56.841648 7234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:56.841998 7237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:56.842058 7235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:56.842820 7229 server_base.cc:1034] running on GCE node
I20260926 15:14:56.843370 7229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:56.844086 7229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:56.845278 7229 hybrid_clock.cc:648] HybridClock initialized: now 1790435696845246 us; error 42 us; skew 500 ppm
I20260926 15:14:56.845534 7229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:56 dist-test-slave-j6q7 krb5kdc[6990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435696, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:56.848091 7229 init.cc:377] Logged in from keytab as kudu/127.6.209.194@KRBTEST.COM (short username kudu)
I20260926 15:14:56.849284 7229 webserver.cc:460] Webserver started at http://127.6.209.194:41333/ using document root <none> and password file <none>
I20260926 15:14:56.849766 7229 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:56.849846 7229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:56.850057 7229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:56.852336 7229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "bbeb3434fbbd436ab90b8bb95ad80941"
format_stamp: "Formatted at 2026-09-26 15:14:56 on dist-test-slave-j6q7"
server_key: "2d92e4706a8fc18bedf7c4a81647a351"
server_key_iv: "05d5526e5d7625026e07c78845015e01"
server_key_version: "encryptionkey@0"
I20260926 15:14:56.853044 7229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "bbeb3434fbbd436ab90b8bb95ad80941"
format_stamp: "Formatted at 2026-09-26 15:14:56 on dist-test-slave-j6q7"
server_key: "2d92e4706a8fc18bedf7c4a81647a351"
server_key_iv: "05d5526e5d7625026e07c78845015e01"
server_key_version: "encryptionkey@0"
I20260926 15:14:56.857141 7229 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:14:56.859947 7244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:56.861088 7229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260926 15:14:56.861229 7229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "bbeb3434fbbd436ab90b8bb95ad80941"
format_stamp: "Formatted at 2026-09-26 15:14:56 on dist-test-slave-j6q7"
server_key: "2d92e4706a8fc18bedf7c4a81647a351"
server_key_iv: "05d5526e5d7625026e07c78845015e01"
server_key_version: "encryptionkey@0"
I20260926 15:14:56.861435 7229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:56.874783 7229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:56.877938 7229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:56.878177 7229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:56.878929 7229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:14:56.879927 7229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:14:56.879993 7229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:56.880132 7229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:14:56.880182 7229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:56.911656 7229 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:44087
I20260926 15:14:56.911721 7356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:44087 every 8 connection(s)
I20260926 15:14:56.912760 7229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:14:56.917532 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 7229
I20260926 15:14:56.917692 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:14:56.918030 6983 external_mini_cluster.cc:1428] Setting key 07b8ce5a40a5eba1c7ddee823c6d897b
Sep 26 15:14:56 dist-test-slave-j6q7 krb5kdc[6990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435696, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:14:56.929636 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:56.915242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:51561 (local address 127.6.209.254:40995)
0926 15:14:56.915400 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:56.915404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:56.916356 (+ 952us) server_negotiation.cc:407] Connection header received
0926 15:14:56.917915 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:56.917921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:56.917999 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:56.918096 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:56.919419 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:56.920055 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:56.921404 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:56.921783 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:56.921905 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:56.924854 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:56.924859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:56.924863 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:56.924893 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:56.927075 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:56.927711 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:56.927717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:56.927720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:56.927829 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:56.928205 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:56.928210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:56.928213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:56.928437 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:56.928585 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:56.929161 (+ 576us) server_negotiation.cc:299] Negotiation successful
0926 15:14:56.929335 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 15:14:56.930362 7359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:56.915640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40995 (local address 127.6.209.194:51561)
0926 15:14:56.916237 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:56.916267 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 15:14:56.917662 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:14:56.918309 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:14:56.918317 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:14:56.918761 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:14:56.919246 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:56.919261 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:56.920187 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:56.920191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:56.921278 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:56.921288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:56.921982 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:56.921989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:56.922238 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:56.922853 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:14:56.922873 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:14:56.924674 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:14:56.927199 (+ 2525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:56.927206 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:56.927218 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:56.927610 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:56.927925 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:56.927930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:56.927934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:56.928101 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:56.928555 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:14:56.928562 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:14:56.928870 (+ 308us) client_negotiation.cc:769] Sending connection context
0926 15:14:56.929099 (+ 229us) client_negotiation.cc:240] Negotiation successful
0926 15:14:56.929473 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":237,"threads_started":1}
WARNING: no policy specified for kudu/127.6.209.195@KRBTEST.COM; defaulting to no policy
I20260926 15:14:56.931942 7357 heartbeater.cc:347] Connected to a master server at 127.6.209.254:40995
I20260926 15:14:56.932224 7357 heartbeater.cc:464] Registering TS with master...
I20260926 15:14:56.932907 7357 heartbeater.cc:511] Master 127.6.209.254:40995 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.195@KRBTEST.COM" created.
I20260926 15:14:56.934157 7035 ts_manager.cc:194] Registered new tserver with Master: bbeb3434fbbd436ab90b8bb95ad80941 (127.6.209.194:44087)
I20260926 15:14:56.934804 7035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.194@KRBTEST.COM'} at 127.6.209.194:51561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
I20260926 15:14:56.949682 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:40995
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:36415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:57.228075 7362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:57.228308 7362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:57.228380 7362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:57.234699 7362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:57.234783 7362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:57.234891 7362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:14:57.243837 7362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:36415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:40995
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:57.244901 7362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:57.245982 7362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:57.257220 7367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:57.257261 7370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:57.257313 7368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:57.258395 7362 server_base.cc:1034] running on GCE node
I20260926 15:14:57.258893 7362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:57.259631 7362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:57.260802 7362 hybrid_clock.cc:648] HybridClock initialized: now 1790435697260756 us; error 43 us; skew 500 ppm
I20260926 15:14:57.261085 7362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:57 dist-test-slave-j6q7 krb5kdc[6990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435697, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:57.263988 7362 init.cc:377] Logged in from keytab as kudu/127.6.209.195@KRBTEST.COM (short username kudu)
I20260926 15:14:57.265280 7362 webserver.cc:460] Webserver started at http://127.6.209.195:45719/ using document root <none> and password file <none>
I20260926 15:14:57.265846 7362 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:57.265962 7362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:57.266234 7362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:57.268532 7362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "69e15653407a4ff0a87ed210a07defde"
format_stamp: "Formatted at 2026-09-26 15:14:57 on dist-test-slave-j6q7"
server_key: "5747d68f88fe3d454a9ae59408afa556"
server_key_iv: "f43ba41c68c7bea6eabca178fc964da4"
server_key_version: "encryptionkey@0"
I20260926 15:14:57.269217 7362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "69e15653407a4ff0a87ed210a07defde"
format_stamp: "Formatted at 2026-09-26 15:14:57 on dist-test-slave-j6q7"
server_key: "5747d68f88fe3d454a9ae59408afa556"
server_key_iv: "f43ba41c68c7bea6eabca178fc964da4"
server_key_version: "encryptionkey@0"
I20260926 15:14:57.273588 7362 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:14:57.276764 7377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:57.277825 7362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 15:14:57.277966 7362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "69e15653407a4ff0a87ed210a07defde"
format_stamp: "Formatted at 2026-09-26 15:14:57 on dist-test-slave-j6q7"
server_key: "5747d68f88fe3d454a9ae59408afa556"
server_key_iv: "f43ba41c68c7bea6eabca178fc964da4"
server_key_version: "encryptionkey@0"
I20260926 15:14:57.278142 7362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:57.299651 7362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:57.303689 7362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:57.303921 7362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:57.304663 7362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:14:57.305713 7362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:14:57.305800 7362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:57.305923 7362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:14:57.305985 7362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:57.337381 7362 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:43539
I20260926 15:14:57.337460 7489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:43539 every 8 connection(s)
I20260926 15:14:57.338522 7362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:14:57.341843 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 7362
I20260926 15:14:57.342015 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:14:57.342474 6983 external_mini_cluster.cc:1428] Setting key 7d6dfca5a2d4176f60b0cfbe22858f7c
Sep 26 15:14:57 dist-test-slave-j6q7 krb5kdc[6990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435697, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:14:57.355753 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:57.340666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:34699 (local address 127.6.209.254:40995)
0926 15:14:57.340904 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:57.340911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:57.341946 (+ 1035us) server_negotiation.cc:407] Connection header received
0926 15:14:57.344072 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:57.344077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:57.344155 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:57.344252 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:57.345977 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:57.346657 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:57.347930 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:57.348244 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:57.348321 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:57.350965 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:57.350969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:57.350972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:57.350997 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:57.353713 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:57.354259 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:57.354262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:57.354264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:57.354354 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:57.354643 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:57.354645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:57.354647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:57.354852 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:57.354941 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:57.355404 (+ 463us) server_negotiation.cc:299] Negotiation successful
0926 15:14:57.355547 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 15:14:57.356941 7492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:57.341161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40995 (local address 127.6.209.195:34699)
0926 15:14:57.341803 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:57.341837 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 15:14:57.343783 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:14:57.344421 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:14:57.344428 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:14:57.345088 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:14:57.345805 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:57.345828 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:57.346767 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:57.346769 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:57.347818 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:57.347827 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:57.348387 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:57.348390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:57.348540 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:57.349072 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:14:57.349089 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:14:57.350800 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:14:57.353803 (+ 3003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:57.353808 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:57.353817 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:57.354158 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:57.354427 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:57.354429 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:57.354431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:57.354563 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:57.354938 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:14:57.354943 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:14:57.355156 (+ 213us) client_negotiation.cc:769] Sending connection context
0926 15:14:57.355327 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 15:14:57.355624 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":236,"threads_started":1}
I20260926 15:14:57.358214 7490 heartbeater.cc:347] Connected to a master server at 127.6.209.254:40995
I20260926 15:14:57.358453 7490 heartbeater.cc:464] Registering TS with master...
I20260926 15:14:57.358992 7490 heartbeater.cc:511] Master 127.6.209.254:40995 requested a full tablet report, sending...
I20260926 15:14:57.360134 7035 ts_manager.cc:194] Registered new tserver with Master: 69e15653407a4ff0a87ed210a07defde (127.6.209.195:43539)
I20260926 15:14:57.360857 7035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.195@KRBTEST.COM'} at 127.6.209.195:34699
I20260926 15:14:57.369707 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 15:14:57 dist-test-slave-j6q7 krb5kdc[6990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435697, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 15:14:57 dist-test-slave-j6q7 krb5kdc[6990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435697, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.209.193@KRBTEST.COM
I20260926 15:14:57.407397 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:57.393742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57240 (local address 127.6.209.193:44101)
0926 15:14:57.394244 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:57.394251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:57.394358 (+ 107us) server_negotiation.cc:407] Connection header received
0926 15:14:57.394540 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:57.394546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:57.394722 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:57.394858 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:57.396131 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:57.397049 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:57.398490 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:57.398926 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:57.399041 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:57.402431 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:57.402440 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:57.402454 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:57.402481 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:57.404873 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:57.405536 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:57.405546 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:57.405554 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:57.405665 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:57.406035 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:57.406042 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:57.406045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:57.406351 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:57.406642 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:57.406873 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 15:14:57.407110 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":234,"threads_started":1}
W20260926 15:14:57.408458 7139 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57240
Sep 26 15:14:57 dist-test-slave-j6q7 krb5kdc[6990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 15:14:57 dist-test-slave-j6q7 krb5kdc[6990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.193@KRBTEST.COM
I20260926 15:14:57.443840 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:57.432061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57244 (local address 127.6.209.193:44101)
0926 15:14:57.432279 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:57.432284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:57.432433 (+ 149us) server_negotiation.cc:407] Connection header received
0926 15:14:57.432619 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:57.432626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:57.432686 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:57.432780 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:57.433971 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:57.434572 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:57.435586 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:57.435922 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:57.436035 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:57.439398 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:57.439403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:57.439405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:57.439428 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:57.441442 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:57.441991 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:57.441997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:57.442005 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:57.442110 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:57.442527 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:57.442533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:57.442536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:57.442818 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:57.443063 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:57.443335 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 15:14:57.443561 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 15:14:57.444967 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 7096
I20260926 15:14:57.454932 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:57.445809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57252 (local address 127.6.209.193:44101)
0926 15:14:57.445992 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:57.446004 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:57.446071 (+ 67us) server_negotiation.cc:407] Connection header received
0926 15:14:57.446169 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:57.446172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:57.446217 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:57.446285 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:57.447192 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:57.447736 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:57.448801 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:57.449184 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:57.449260 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:57.450891 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:57.450894 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:57.450897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:57.450918 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:57.453098 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:57.453589 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:57.453592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:57.453594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:57.453683 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:57.453919 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:57.453922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:57.453924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:57.454108 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:57.454252 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:57.454458 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 15:14:57.454587 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 15:14:57.455883 7219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:14:57.529044 7224 heartbeater.cc:503] Master 127.6.209.254:40995 was elected leader, sending a full tablet report...
I20260926 15:14:57.529600 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 7096
W20260926 15:14:57.543715 7021 connection.cc:535] server connection from 127.6.209.193:57499 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 15:14:57.544035 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 7229
Sep 26 15:14:57 dist-test-slave-j6q7 krb5kdc[6990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.194@KRBTEST.COM
I20260926 15:14:57.557222 7504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:57.544753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47596 (local address 127.6.209.194:44087)
0926 15:14:57.545207 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:57.545213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:57.545233 (+ 20us) server_negotiation.cc:407] Connection header received
0926 15:14:57.545280 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:57.545285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:57.545467 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:57.545611 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:57.546619 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:57.547519 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:57.548534 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:57.549060 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:57.549190 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:57.552355 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:57.552364 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:57.552377 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:57.552406 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:57.554738 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:57.555276 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:57.555282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:57.555285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:57.555412 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:57.555768 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:57.555773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:57.555777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:57.556074 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:57.556384 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:57.556669 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 15:14:57.556908 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":246,"threads_started":1}
I20260926 15:14:57.558266 7352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:14:57.630330 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 7229
I20260926 15:14:57.650489 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 7362
Sep 26 15:14:57 dist-test-slave-j6q7 krb5kdc[6990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.195@KRBTEST.COM
I20260926 15:14:57.664062 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:57.651399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55872 (local address 127.6.209.195:43539)
0926 15:14:57.651968 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:57.651974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:57.651998 (+ 24us) server_negotiation.cc:407] Connection header received
0926 15:14:57.652063 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:57.652068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:57.652308 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:57.652433 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:57.653628 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:57.654452 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:57.655461 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:57.655913 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:57.655993 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:57.659491 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:57.659499 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:57.659512 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:57.659540 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:57.661916 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:57.662457 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:57.662463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:57.662465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:57.662566 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:57.662842 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:57.662845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:57.662847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:57.663141 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:57.663355 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:57.663553 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 15:14:57.663713 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":315,"threads_started":1}
I20260926 15:14:57.665165 7485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:14:57.737709 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 7362
I20260926 15:14:57.751924 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 7003
I20260926 15:14:57.753012 7065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:14:57.842775 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 7003
2026-09-26T15:14:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3384 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:14:57 dist-test-slave-j6q7 krb5kdc[7511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:14:57 dist-test-slave-j6q7 krb5kdc[7511](info): set up 2 sockets
Sep 26 15:14:57 dist-test-slave-j6q7 krb5kdc[7511](info): commencing operation
krb5kdc: starting...
W20260926 15:14:58.812194 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.002s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:14:58 dist-test-slave-j6q7 krb5kdc[7511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:14:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:14:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:14:58.957629 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:35335
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:35335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:59.231369 7524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:59.231647 7524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:59.231792 7524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:59.238310 7524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:59.238421 7524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:59.238510 7524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:59.238596 7524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:59.238670 7524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:59.247181 7524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:35335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:35335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:59.248297 7524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:59.249433 7524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:59.259641 7529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:59.259719 7532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:59.259702 7530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:59.260330 7524 server_base.cc:1034] running on GCE node
I20260926 15:14:59.261081 7524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:59.262274 7524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:59.263549 7524 hybrid_clock.cc:648] HybridClock initialized: now 1790435699263505 us; error 47 us; skew 500 ppm
I20260926 15:14:59.263845 7524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:59 dist-test-slave-j6q7 krb5kdc[7511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435699, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:59.266422 7524 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:14:59.267585 7524 webserver.cc:460] Webserver started at http://127.6.209.254:46577/ using document root <none> and password file <none>
I20260926 15:14:59.268157 7524 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:59.268275 7524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:59.268523 7524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:59.270957 7524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "0b9dec44f23c45daa7c6fb575c56aeba"
format_stamp: "Formatted at 2026-09-26 15:14:59 on dist-test-slave-j6q7"
server_key: "5b5f714e0d37f7b2bfcfee14f80d4bf5"
server_key_iv: "b7535bf6f750873df72d2340bf866791"
server_key_version: "encryptionkey@0"
I20260926 15:14:59.271605 7524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "0b9dec44f23c45daa7c6fb575c56aeba"
format_stamp: "Formatted at 2026-09-26 15:14:59 on dist-test-slave-j6q7"
server_key: "5b5f714e0d37f7b2bfcfee14f80d4bf5"
server_key_iv: "b7535bf6f750873df72d2340bf866791"
server_key_version: "encryptionkey@0"
I20260926 15:14:59.276008 7524 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 15:14:59.278699 7539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:59.279721 7524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:14:59.279874 7524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "0b9dec44f23c45daa7c6fb575c56aeba"
format_stamp: "Formatted at 2026-09-26 15:14:59 on dist-test-slave-j6q7"
server_key: "5b5f714e0d37f7b2bfcfee14f80d4bf5"
server_key_iv: "b7535bf6f750873df72d2340bf866791"
server_key_version: "encryptionkey@0"
I20260926 15:14:59.280043 7524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:59.293831 7524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:59.297183 7524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:59.297442 7524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:59.313061 7524 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:35335
I20260926 15:14:59.313090 7590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:35335 every 8 connection(s)
I20260926 15:14:59.314178 7524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:14:59.317206 7591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:59.320497 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 7524
I20260926 15:14:59.320720 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:14:59.321496 6983 external_mini_cluster.cc:1428] Setting key 71755b64271ddd9895e5c43ed22761df
I20260926 15:14:59.327505 7591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b9dec44f23c45daa7c6fb575c56aeba: Bootstrap starting.
I20260926 15:14:59.330633 7591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b9dec44f23c45daa7c6fb575c56aeba: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:59.331666 7591 log.cc:824] T 00000000000000000000000000000000 P 0b9dec44f23c45daa7c6fb575c56aeba: Log is configured to *not* fsync() on all Append() calls
I20260926 15:14:59.334106 7591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b9dec44f23c45daa7c6fb575c56aeba: No bootstrap required, opened a new log
Sep 26 15:14:59 dist-test-slave-j6q7 krb5kdc[7511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:14:59.343185 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:59.323389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40456 (local address 127.6.209.254:35335)
0926 15:14:59.324096 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:59.324106 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:59.324142 (+ 36us) server_negotiation.cc:407] Connection header received
0926 15:14:59.326179 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:59.326198 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:59.326513 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:59.326792 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:59.328400 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:59.329526 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:59.331184 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:59.331827 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:59.331955 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:59.335652 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:59.335665 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:59.335679 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:59.335721 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:59.339189 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:59.339757 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:59.339764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:59.339771 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:59.339947 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:59.340258 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:59.340262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:59.340265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:59.340918 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:59.341185 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:59.341579 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 15:14:59.341976 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":345,"threads_started":1}
I20260926 15:14:59.343743 7591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b9dec44f23c45daa7c6fb575c56aeba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9dec44f23c45daa7c6fb575c56aeba" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35335 } }
I20260926 15:14:59.344146 7591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b9dec44f23c45daa7c6fb575c56aeba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:14:59.344262 7591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b9dec44f23c45daa7c6fb575c56aeba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b9dec44f23c45daa7c6fb575c56aeba, State: Initialized, Role: FOLLOWER
I20260926 15:14:59.344867 7591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b9dec44f23c45daa7c6fb575c56aeba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9dec44f23c45daa7c6fb575c56aeba" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35335 } }
I20260926 15:14:59.345185 7591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b9dec44f23c45daa7c6fb575c56aeba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:14:59.345369 7591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b9dec44f23c45daa7c6fb575c56aeba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:14:59.345532 7591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b9dec44f23c45daa7c6fb575c56aeba [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:59.347424 7591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b9dec44f23c45daa7c6fb575c56aeba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9dec44f23c45daa7c6fb575c56aeba" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35335 } }
I20260926 15:14:59.347824 7591 leader_election.cc:302] T 00000000000000000000000000000000 P 0b9dec44f23c45daa7c6fb575c56aeba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b9dec44f23c45daa7c6fb575c56aeba; no voters:
I20260926 15:14:59.348268 7591 leader_election.cc:288] T 00000000000000000000000000000000 P 0b9dec44f23c45daa7c6fb575c56aeba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:14:59.348392 7596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b9dec44f23c45daa7c6fb575c56aeba [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:14:59.348989 7596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b9dec44f23c45daa7c6fb575c56aeba [term 1 LEADER]: Becoming Leader. State: Replica: 0b9dec44f23c45daa7c6fb575c56aeba, State: Running, Role: LEADER
I20260926 15:14:59.349479 7596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b9dec44f23c45daa7c6fb575c56aeba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9dec44f23c45daa7c6fb575c56aeba" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35335 } }
I20260926 15:14:59.349993 7591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b9dec44f23c45daa7c6fb575c56aeba [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:14:59.351920 7598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b9dec44f23c45daa7c6fb575c56aeba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b9dec44f23c45daa7c6fb575c56aeba. Latest consensus state: current_term: 1 leader_uuid: "0b9dec44f23c45daa7c6fb575c56aeba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9dec44f23c45daa7c6fb575c56aeba" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35335 } } }
I20260926 15:14:59.352240 7598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b9dec44f23c45daa7c6fb575c56aeba [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:59.351888 7597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b9dec44f23c45daa7c6fb575c56aeba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b9dec44f23c45daa7c6fb575c56aeba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9dec44f23c45daa7c6fb575c56aeba" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35335 } } }
I20260926 15:14:59.352747 7597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b9dec44f23c45daa7c6fb575c56aeba [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:59.357928 7602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:14:59.362565 7602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:14:59.370738 7602 catalog_manager.cc:1347] Generated new cluster ID: cd5c96e3805d4a79bbf2828cbc5f66da
I20260926 15:14:59.370829 7602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:14:59.389881 7602 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:14:59.390904 7602 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:14:59.395804 7602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b9dec44f23c45daa7c6fb575c56aeba: Generated new TSK 0
I20260926 15:14:59.396466 7602 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
I20260926 15:14:59.443785 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35335
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:59.752143 7617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:59.752408 7617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:59.752616 7617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:59.759071 7617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:59.759157 7617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:59.759346 7617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:14:59.769261 7617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35335
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:59.770319 7617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:59.771600 7617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:59.784191 7622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:59.784386 7625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:59.784170 7623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:59.785015 7617 server_base.cc:1034] running on GCE node
I20260926 15:14:59.785531 7617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:59.786319 7617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:59.787526 7617 hybrid_clock.cc:648] HybridClock initialized: now 1790435699787482 us; error 44 us; skew 500 ppm
I20260926 15:14:59.787782 7617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:59 dist-test-slave-j6q7 krb5kdc[7511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435699, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:59.795344 7617 init.cc:377] Logged in from keytab as kudu/127.6.209.193@KRBTEST.COM (short username kudu)
I20260926 15:14:59.796691 7617 webserver.cc:460] Webserver started at http://127.6.209.193:42887/ using document root <none> and password file <none>
I20260926 15:14:59.797283 7617 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:59.797405 7617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:59.797665 7617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:59.799957 7617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "d735da26ef5446078f2136fbfca8a255"
format_stamp: "Formatted at 2026-09-26 15:14:59 on dist-test-slave-j6q7"
server_key: "44cb9b4401384800cca80a912ed52e08"
server_key_iv: "713b89aa231e81bfc431134ee118d84d"
server_key_version: "encryptionkey@0"
I20260926 15:14:59.800645 7617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "d735da26ef5446078f2136fbfca8a255"
format_stamp: "Formatted at 2026-09-26 15:14:59 on dist-test-slave-j6q7"
server_key: "44cb9b4401384800cca80a912ed52e08"
server_key_iv: "713b89aa231e81bfc431134ee118d84d"
server_key_version: "encryptionkey@0"
I20260926 15:14:59.805204 7617 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 15:14:59.808172 7632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:59.809303 7617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:14:59.809453 7617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "d735da26ef5446078f2136fbfca8a255"
format_stamp: "Formatted at 2026-09-26 15:14:59 on dist-test-slave-j6q7"
server_key: "44cb9b4401384800cca80a912ed52e08"
server_key_iv: "713b89aa231e81bfc431134ee118d84d"
server_key_version: "encryptionkey@0"
I20260926 15:14:59.809628 7617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:59.821619 7617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:59.825062 7617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:59.825296 7617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:59.826076 7617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:14:59.827237 7617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:14:59.827329 7617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:59.827469 7617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:14:59.827535 7617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:59.858357 7617 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:36233
I20260926 15:14:59.858388 7744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:36233 every 8 connection(s)
I20260926 15:14:59.859488 7617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:14:59.867621 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 7617
I20260926 15:14:59.867833 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:14:59.868556 6983 external_mini_cluster.cc:1428] Setting key 6ee1b16e2b12622ae68220bb04ff0422
Sep 26 15:14:59 dist-test-slave-j6q7 krb5kdc[7511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435699, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:14:59.880300 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:59.861849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:34395 (local address 127.6.209.254:35335)
0926 15:14:59.862242 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:59.862246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:59.863144 (+ 898us) server_negotiation.cc:407] Connection header received
0926 15:14:59.865230 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:59.865238 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:59.865351 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:59.865494 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:59.867316 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:59.868435 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:59.870566 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:59.871108 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:59.871227 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:59.875202 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:59.875208 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:59.875212 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:59.875247 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:59.877443 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:59.878233 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:59.878236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:59.878239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:59.878346 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:59.878787 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:59.878792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:59.878795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:59.879071 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:59.879206 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:59.879942 (+ 736us) server_negotiation.cc:299] Negotiation successful
0926 15:14:59.880078 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":216,"threads_started":1}
I20260926 15:14:59.881573 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:59.862327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:35335 (local address 127.6.209.193:34395)
0926 15:14:59.862989 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:59.863026 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 15:14:59.864895 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:14:59.865724 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:14:59.865736 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:14:59.866381 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:14:59.867091 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:59.867116 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:59.868597 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:59.868602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:59.870384 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:59.870400 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:59.871524 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:59.871533 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:59.871830 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:59.872520 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:14:59.872558 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:14:59.875028 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:14:59.877561 (+ 2533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:59.877570 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:59.877585 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:59.878111 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:59.878439 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:59.878446 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:59.878452 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:59.878662 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:59.879216 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:14:59.879225 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:14:59.879583 (+ 358us) client_negotiation.cc:769] Sending connection context
0926 15:14:59.879830 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 15:14:59.880282 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":252,"threads_started":1}
I20260926 15:14:59.883194 7745 heartbeater.cc:347] Connected to a master server at 127.6.209.254:35335
I20260926 15:14:59.883502 7745 heartbeater.cc:464] Registering TS with master...
I20260926 15:14:59.884344 7745 heartbeater.cc:511] Master 127.6.209.254:35335 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.209.194@KRBTEST.COM; defaulting to no policy
I20260926 15:14:59.886351 7556 ts_manager.cc:194] Registered new tserver with Master: d735da26ef5446078f2136fbfca8a255 (127.6.209.193:36233)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.194@KRBTEST.COM" created.
I20260926 15:14:59.888164 7556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.193@KRBTEST.COM'} at 127.6.209.193:34395
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
I20260926 15:14:59.905803 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35335
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:00.188163 7751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:00.188380 7751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:00.188455 7751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:00.194715 7751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:00.194794 7751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:00.194887 7751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:15:00.203459 7751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35335
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:00.204391 7751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:00.205449 7751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:00.216651 7757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:00.216745 7756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:00.216754 7759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:00.217712 7751 server_base.cc:1034] running on GCE node
I20260926 15:15:00.218257 7751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:00.219064 7751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:00.220248 7751 hybrid_clock.cc:648] HybridClock initialized: now 1790435700220199 us; error 46 us; skew 500 ppm
I20260926 15:15:00.220489 7751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:00 dist-test-slave-j6q7 krb5kdc[7511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435700, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:00.223377 7751 init.cc:377] Logged in from keytab as kudu/127.6.209.194@KRBTEST.COM (short username kudu)
I20260926 15:15:00.224643 7751 webserver.cc:460] Webserver started at http://127.6.209.194:37971/ using document root <none> and password file <none>
I20260926 15:15:00.225275 7751 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:00.225389 7751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:00.225670 7751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:00.228081 7751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "9cfb1cde0d0741f69ef6788baae94dbc"
format_stamp: "Formatted at 2026-09-26 15:15:00 on dist-test-slave-j6q7"
server_key: "d849d6709efafd39c1c5642ef71f1504"
server_key_iv: "7c9085313766c6ff5a2e19e3bde367e1"
server_key_version: "encryptionkey@0"
I20260926 15:15:00.228735 7751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "9cfb1cde0d0741f69ef6788baae94dbc"
format_stamp: "Formatted at 2026-09-26 15:15:00 on dist-test-slave-j6q7"
server_key: "d849d6709efafd39c1c5642ef71f1504"
server_key_iv: "7c9085313766c6ff5a2e19e3bde367e1"
server_key_version: "encryptionkey@0"
I20260926 15:15:00.233419 7751 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 15:15:00.236191 7766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:00.237299 7751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260926 15:15:00.237449 7751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "9cfb1cde0d0741f69ef6788baae94dbc"
format_stamp: "Formatted at 2026-09-26 15:15:00 on dist-test-slave-j6q7"
server_key: "d849d6709efafd39c1c5642ef71f1504"
server_key_iv: "7c9085313766c6ff5a2e19e3bde367e1"
server_key_version: "encryptionkey@0"
I20260926 15:15:00.237619 7751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:00.250056 7751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:00.253893 7751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:00.254130 7751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:00.254921 7751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:00.256043 7751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:00.256130 7751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:00.256258 7751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:00.256319 7751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:00.284672 7751 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:45985
I20260926 15:15:00.284741 7878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:45985 every 8 connection(s)
I20260926 15:15:00.285804 7751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:15:00.287753 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 7751
I20260926 15:15:00.287935 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:15:00.288303 6983 external_mini_cluster.cc:1428] Setting key f263fc5ab4d0d713ebef4e04dd353f2e
Sep 26 15:15:00 dist-test-slave-j6q7 krb5kdc[7511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435700, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.209.195@KRBTEST.COM; defaulting to no policy
I20260926 15:15:00.304286 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:00.288901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:50327 (local address 127.6.209.254:35335)
0926 15:15:00.289055 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:00.289061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:00.289089 (+ 28us) server_negotiation.cc:407] Connection header received
0926 15:15:00.291157 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:00.291161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:00.291225 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:00.291296 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:00.293212 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.293869 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.295382 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.295766 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.295889 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:00.298934 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:00.298939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:00.298942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:00.298968 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:00.301405 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:00.302111 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:00.302118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:00.302124 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:00.302293 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:00.302718 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:00.302723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:00.302727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:00.302995 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:00.303166 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:00.303759 (+ 593us) server_negotiation.cc:299] Negotiation successful
0926 15:15:00.303984 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.195@KRBTEST.COM" created.
I20260926 15:15:00.305287 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:00.288241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:35335 (local address 127.6.209.194:50327)
0926 15:15:00.288967 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:00.289010 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:00.290875 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:00.291530 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:00.291544 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:00.292237 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:00.293053 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:00.293079 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.294006 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.294013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:00.295258 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:00.295268 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.295959 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.295966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:00.296169 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:00.296787 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:00.296843 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:00.298752 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:00.301538 (+ 2786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:00.301548 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:00.301561 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:00.301951 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:00.302400 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:00.302408 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:00.302412 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:00.302603 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:00.303141 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:00.303148 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:00.303435 (+ 287us) client_negotiation.cc:769] Sending connection context
0926 15:15:00.303679 (+ 244us) client_negotiation.cc:240] Negotiation successful
0926 15:15:00.304098 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":317,"threads_started":1}
I20260926 15:15:00.306573 7879 heartbeater.cc:347] Connected to a master server at 127.6.209.254:35335
I20260926 15:15:00.306845 7879 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:00.307605 7879 heartbeater.cc:511] Master 127.6.209.254:35335 requested a full tablet report, sending...
I20260926 15:15:00.309000 7556 ts_manager.cc:194] Registered new tserver with Master: 9cfb1cde0d0741f69ef6788baae94dbc (127.6.209.194:45985)
I20260926 15:15:00.309708 7556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.194@KRBTEST.COM'} at 127.6.209.194:50327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
I20260926 15:15:00.327533 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35335
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:00.626639 7884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:00.626916 7884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:00.627092 7884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:00.634397 7884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:00.634526 7884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:00.634716 7884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:15:00.647151 7884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35335
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:00.649075 7884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:00.650717 7884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:00.664352 7889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:00.664384 7890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:00.664328 7892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:00.665217 7884 server_base.cc:1034] running on GCE node
I20260926 15:15:00.665704 7884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:00.666524 7884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:00.667737 7884 hybrid_clock.cc:648] HybridClock initialized: now 1790435700667698 us; error 38 us; skew 500 ppm
I20260926 15:15:00.668012 7884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:00 dist-test-slave-j6q7 krb5kdc[7511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435700, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:00.670783 7884 init.cc:377] Logged in from keytab as kudu/127.6.209.195@KRBTEST.COM (short username kudu)
I20260926 15:15:00.671990 7884 webserver.cc:460] Webserver started at http://127.6.209.195:42895/ using document root <none> and password file <none>
I20260926 15:15:00.672606 7884 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:00.672744 7884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:00.673045 7884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:00.675514 7884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "9e349a2983b24063941df6c84bdda4dd"
format_stamp: "Formatted at 2026-09-26 15:15:00 on dist-test-slave-j6q7"
server_key: "7070603b4a6ae9f53656dac1328d9dc1"
server_key_iv: "e70f0f05d3a7565a55811622c4f9b0ad"
server_key_version: "encryptionkey@0"
I20260926 15:15:00.676285 7884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "9e349a2983b24063941df6c84bdda4dd"
format_stamp: "Formatted at 2026-09-26 15:15:00 on dist-test-slave-j6q7"
server_key: "7070603b4a6ae9f53656dac1328d9dc1"
server_key_iv: "e70f0f05d3a7565a55811622c4f9b0ad"
server_key_version: "encryptionkey@0"
I20260926 15:15:00.680680 7884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 15:15:00.683918 7899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:00.685639 7884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:15:00.685807 7884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "9e349a2983b24063941df6c84bdda4dd"
format_stamp: "Formatted at 2026-09-26 15:15:00 on dist-test-slave-j6q7"
server_key: "7070603b4a6ae9f53656dac1328d9dc1"
server_key_iv: "e70f0f05d3a7565a55811622c4f9b0ad"
server_key_version: "encryptionkey@0"
I20260926 15:15:00.685971 7884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:00.699222 7884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:00.702632 7884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:00.702836 7884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:00.703678 7884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:00.704895 7884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:00.704962 7884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:00.705133 7884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:00.705185 7884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:00.734815 7884 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:32773
I20260926 15:15:00.734858 8011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:32773 every 8 connection(s)
I20260926 15:15:00.736013 7884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:15:00.741874 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 7884
I20260926 15:15:00.742063 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:15:00.742602 6983 external_mini_cluster.cc:1428] Setting key 5a5a4a116040c3df1c7cf0eb18a7b7eb
Sep 26 15:15:00 dist-test-slave-j6q7 krb5kdc[7511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435700, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:00.755141 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:00.738419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:36713 (local address 127.6.209.254:35335)
0926 15:15:00.738622 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:00.738628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:00.740025 (+ 1397us) server_negotiation.cc:407] Connection header received
0926 15:15:00.742108 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:00.742112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:00.742217 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:00.742300 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:00.744177 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.744994 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.746553 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.746963 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.747086 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:00.750291 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:00.750296 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:00.750299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:00.750361 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:00.752777 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:00.753424 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:00.753428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:00.753430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:00.753538 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:00.753899 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:00.753901 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:00.753903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:00.754140 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:00.754239 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:00.754762 (+ 523us) server_negotiation.cc:299] Negotiation successful
0926 15:15:00.754898 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 15:15:00.756071 8014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:00.738867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:35335 (local address 127.6.209.195:36713)
0926 15:15:00.739875 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:00.739913 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:00.741819 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:00.742554 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:00.742564 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:00.743232 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:00.744011 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:00.744037 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.745176 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.745181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:00.746407 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:00.746418 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.747182 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.747190 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:00.747404 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:00.747965 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:00.747984 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:00.750123 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:00.752933 (+ 2810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:00.752939 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:00.752949 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:00.753329 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:00.753638 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:00.753641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:00.753643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:00.753815 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:00.754234 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:00.754239 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:00.754489 (+ 250us) client_negotiation.cc:769] Sending connection context
0926 15:15:00.754686 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 15:15:00.755012 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":629,"threads_started":1}
I20260926 15:15:00.757419 8012 heartbeater.cc:347] Connected to a master server at 127.6.209.254:35335
I20260926 15:15:00.757733 8012 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:00.758380 8012 heartbeater.cc:511] Master 127.6.209.254:35335 requested a full tablet report, sending...
I20260926 15:15:00.759691 7555 ts_manager.cc:194] Registered new tserver with Master: 9e349a2983b24063941df6c84bdda4dd (127.6.209.195:32773)
I20260926 15:15:00.760536 7555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.195@KRBTEST.COM'} at 127.6.209.195:36713
I20260926 15:15:00.769048 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:15:00.785562 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:00.775026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40460 (local address 127.6.209.254:35335)
0926 15:15:00.775200 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:00.775206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:00.775664 (+ 458us) server_negotiation.cc:407] Connection header received
0926 15:15:00.775776 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:00.775779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:00.775829 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:00.775897 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:00.777154 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.777730 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.778735 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.779052 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.779129 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:00.780960 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:00.780963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:00.780966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:00.780990 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:00.783430 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:00.783999 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:00.784002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:00.784003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:00.784101 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:00.784434 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:00.784436 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:00.784438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:00.784677 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:00.784881 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:00.785124 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 15:15:00.785266 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 15:15:00.791131 7555 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:15:00.797181 7555 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:15:00.831064 8027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:00.808215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49432 (local address 127.6.209.194:45985)
0926 15:15:00.820058 (+ 11843us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:00.820067 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:00.821389 (+ 1322us) server_negotiation.cc:407] Connection header received
0926 15:15:00.821661 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:00.821669 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:00.821867 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:00.822096 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:00.823136 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.825121 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.828214 (+ 3093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.829592 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.829715 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:00.829888 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:00.830649 (+ 761us) server_negotiation.cc:299] Negotiation successful
0926 15:15:00.830792 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11679,"thread_start_us":11530,"threads_started":1}
I20260926 15:15:00.831190 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:00.807876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58748 (local address 127.6.209.193:36233)
0926 15:15:00.820722 (+ 12846us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:00.820728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:00.820750 (+ 22us) server_negotiation.cc:407] Connection header received
0926 15:15:00.820868 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:00.820879 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:00.821135 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:00.821384 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:00.822254 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.824324 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.828550 (+ 4226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.830257 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.830385 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:00.830551 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:00.830762 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 15:15:00.830897 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12348,"thread_start_us":12228,"threads_started":1}
I20260926 15:15:00.832010 8024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:00.807972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:45985 (local address 127.0.0.1:49432)
0926 15:15:00.821270 (+ 13298us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:00.821289 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:00.821479 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:00.822376 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:00.822383 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:00.822416 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:00.822907 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:00.822921 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.825300 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.825308 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:00.828047 (+ 2739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:00.828058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.829790 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.829796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:00.830275 (+ 479us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:00.830305 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 15:15:00.830439 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 15:15:00.831505 (+ 1066us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13113,"mutex_wait_us":22,"thread_start_us":12749,"threads_started":1}
I20260926 15:15:00.832036 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:00.807668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:36233 (local address 127.0.0.1:58748)
0926 15:15:00.820301 (+ 12633us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:00.820344 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:00.820520 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:00.821618 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:00.821626 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:00.821659 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:00.822018 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:00.822032 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.824529 (+ 2497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.824536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:00.828393 (+ 3857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:00.828405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.830441 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.830447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:00.830577 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:00.830604 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 15:15:00.831409 (+ 805us) client_negotiation.cc:240] Negotiation successful
0926 15:15:00.831613 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12374,"mutex_wait_us":291,"thread_start_us":13290,"threads_started":1}
I20260926 15:15:00.835407 7814 tablet_service.cc:1467] Processing CreateTablet for tablet 71ee2d5703dd4c1f807c103533c438d9 (DEFAULT_TABLE table=test-table [id=86283d705c784d0c9861afdedbde1698]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:00.835430 7680 tablet_service.cc:1467] Processing CreateTablet for tablet 71ee2d5703dd4c1f807c103533c438d9 (DEFAULT_TABLE table=test-table [id=86283d705c784d0c9861afdedbde1698]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:00.835979 8028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:00.821719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:32773 (local address 127.0.0.1:46062)
0926 15:15:00.828127 (+ 6408us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:00.828147 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:00.828271 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:00.829437 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:00.829442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:00.829464 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:00.830228 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:00.830239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.831862 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.831867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:00.833834 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:00.833851 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.835380 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.835387 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:00.835522 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:00.835548 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 15:15:00.835635 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 15:15:00.835725 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6242,"thread_start_us":5960,"threads_started":1}
I20260926 15:15:00.836087 8029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:00.821806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46062 (local address 127.6.209.195:32773)
0926 15:15:00.828939 (+ 7133us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:00.828947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:00.828974 (+ 27us) server_negotiation.cc:407] Connection header received
0926 15:15:00.829058 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:00.829065 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:00.829248 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:00.829435 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:00.830435 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.831732 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.834021 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.835206 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.835319 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:00.835486 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:00.835686 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 15:15:00.835809 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6976,"thread_start_us":6837,"threads_started":1}
I20260926 15:15:00.836562 7814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71ee2d5703dd4c1f807c103533c438d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:00.836619 7680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71ee2d5703dd4c1f807c103533c438d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:00.838860 7947 tablet_service.cc:1467] Processing CreateTablet for tablet 71ee2d5703dd4c1f807c103533c438d9 (DEFAULT_TABLE table=test-table [id=86283d705c784d0c9861afdedbde1698]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:00.840370 7947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71ee2d5703dd4c1f807c103533c438d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:00.846863 8030 tablet_bootstrap.cc:492] T 71ee2d5703dd4c1f807c103533c438d9 P d735da26ef5446078f2136fbfca8a255: Bootstrap starting.
I20260926 15:15:00.849498 8031 tablet_bootstrap.cc:492] T 71ee2d5703dd4c1f807c103533c438d9 P 9cfb1cde0d0741f69ef6788baae94dbc: Bootstrap starting.
I20260926 15:15:00.850011 8030 tablet_bootstrap.cc:654] T 71ee2d5703dd4c1f807c103533c438d9 P d735da26ef5446078f2136fbfca8a255: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:00.851125 8030 log.cc:824] T 71ee2d5703dd4c1f807c103533c438d9 P d735da26ef5446078f2136fbfca8a255: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:00.852427 8031 tablet_bootstrap.cc:654] T 71ee2d5703dd4c1f807c103533c438d9 P 9cfb1cde0d0741f69ef6788baae94dbc: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:00.853248 8033 tablet_bootstrap.cc:492] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd: Bootstrap starting.
I20260926 15:15:00.854032 8031 log.cc:824] T 71ee2d5703dd4c1f807c103533c438d9 P 9cfb1cde0d0741f69ef6788baae94dbc: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:00.854415 8030 tablet_bootstrap.cc:492] T 71ee2d5703dd4c1f807c103533c438d9 P d735da26ef5446078f2136fbfca8a255: No bootstrap required, opened a new log
I20260926 15:15:00.854657 8030 ts_tablet_manager.cc:1397] T 71ee2d5703dd4c1f807c103533c438d9 P d735da26ef5446078f2136fbfca8a255: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260926 15:15:00.855845 8033 tablet_bootstrap.cc:654] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:00.856624 8033 log.cc:824] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:00.856861 8031 tablet_bootstrap.cc:492] T 71ee2d5703dd4c1f807c103533c438d9 P 9cfb1cde0d0741f69ef6788baae94dbc: No bootstrap required, opened a new log
I20260926 15:15:00.857110 8031 ts_tablet_manager.cc:1397] T 71ee2d5703dd4c1f807c103533c438d9 P 9cfb1cde0d0741f69ef6788baae94dbc: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260926 15:15:00.865209 8033 tablet_bootstrap.cc:492] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd: No bootstrap required, opened a new log
I20260926 15:15:00.865196 8030 raft_consensus.cc:348] T 71ee2d5703dd4c1f807c103533c438d9 P d735da26ef5446078f2136fbfca8a255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d735da26ef5446078f2136fbfca8a255" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 36233 } } peers { permanent_uuid: "9cfb1cde0d0741f69ef6788baae94dbc" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 45985 } } peers { permanent_uuid: "9e349a2983b24063941df6c84bdda4dd" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 32773 } }
I20260926 15:15:00.865454 8033 ts_tablet_manager.cc:1397] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20260926 15:15:00.865646 8030 raft_consensus.cc:374] T 71ee2d5703dd4c1f807c103533c438d9 P d735da26ef5446078f2136fbfca8a255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:00.865778 8030 raft_consensus.cc:729] T 71ee2d5703dd4c1f807c103533c438d9 P d735da26ef5446078f2136fbfca8a255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d735da26ef5446078f2136fbfca8a255, State: Initialized, Role: FOLLOWER
I20260926 15:15:00.866581 8030 consensus_queue.cc:260] T 71ee2d5703dd4c1f807c103533c438d9 P d735da26ef5446078f2136fbfca8a255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d735da26ef5446078f2136fbfca8a255" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 36233 } } peers { permanent_uuid: "9cfb1cde0d0741f69ef6788baae94dbc" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 45985 } } peers { permanent_uuid: "9e349a2983b24063941df6c84bdda4dd" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 32773 } }
I20260926 15:15:00.867039 8031 raft_consensus.cc:348] T 71ee2d5703dd4c1f807c103533c438d9 P 9cfb1cde0d0741f69ef6788baae94dbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d735da26ef5446078f2136fbfca8a255" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 36233 } } peers { permanent_uuid: "9cfb1cde0d0741f69ef6788baae94dbc" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 45985 } } peers { permanent_uuid: "9e349a2983b24063941df6c84bdda4dd" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 32773 } }
I20260926 15:15:00.867547 8031 raft_consensus.cc:374] T 71ee2d5703dd4c1f807c103533c438d9 P 9cfb1cde0d0741f69ef6788baae94dbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:00.867736 8031 raft_consensus.cc:729] T 71ee2d5703dd4c1f807c103533c438d9 P 9cfb1cde0d0741f69ef6788baae94dbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cfb1cde0d0741f69ef6788baae94dbc, State: Initialized, Role: FOLLOWER
I20260926 15:15:00.868525 8031 consensus_queue.cc:260] T 71ee2d5703dd4c1f807c103533c438d9 P 9cfb1cde0d0741f69ef6788baae94dbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d735da26ef5446078f2136fbfca8a255" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 36233 } } peers { permanent_uuid: "9cfb1cde0d0741f69ef6788baae94dbc" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 45985 } } peers { permanent_uuid: "9e349a2983b24063941df6c84bdda4dd" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 32773 } }
I20260926 15:15:00.874228 7745 heartbeater.cc:503] Master 127.6.209.254:35335 was elected leader, sending a full tablet report...
I20260926 15:15:00.875051 8030 ts_tablet_manager.cc:1428] T 71ee2d5703dd4c1f807c103533c438d9 P d735da26ef5446078f2136fbfca8a255: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
I20260926 15:15:00.875130 8033 raft_consensus.cc:348] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d735da26ef5446078f2136fbfca8a255" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 36233 } } peers { permanent_uuid: "9cfb1cde0d0741f69ef6788baae94dbc" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 45985 } } peers { permanent_uuid: "9e349a2983b24063941df6c84bdda4dd" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 32773 } }
I20260926 15:15:00.875530 8033 raft_consensus.cc:374] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:00.875663 8033 raft_consensus.cc:729] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e349a2983b24063941df6c84bdda4dd, State: Initialized, Role: FOLLOWER
I20260926 15:15:00.876235 8033 consensus_queue.cc:260] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d735da26ef5446078f2136fbfca8a255" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 36233 } } peers { permanent_uuid: "9cfb1cde0d0741f69ef6788baae94dbc" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 45985 } } peers { permanent_uuid: "9e349a2983b24063941df6c84bdda4dd" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 32773 } }
I20260926 15:15:00.877976 8031 ts_tablet_manager.cc:1428] T 71ee2d5703dd4c1f807c103533c438d9 P 9cfb1cde0d0741f69ef6788baae94dbc: Time spent starting tablet: real 0.021s user 0.014s sys 0.003s
I20260926 15:15:00.878063 7879 heartbeater.cc:503] Master 127.6.209.254:35335 was elected leader, sending a full tablet report...
I20260926 15:15:00.879316 8033 ts_tablet_manager.cc:1428] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 15:15:00.879390 8012 heartbeater.cc:503] Master 127.6.209.254:35335 was elected leader, sending a full tablet report...
I20260926 15:15:00.978751 8038 raft_consensus.cc:482] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:15:00.979038 8038 raft_consensus.cc:504] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d735da26ef5446078f2136fbfca8a255" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 36233 } } peers { permanent_uuid: "9cfb1cde0d0741f69ef6788baae94dbc" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 45985 } } peers { permanent_uuid: "9e349a2983b24063941df6c84bdda4dd" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 32773 } }
I20260926 15:15:00.980609 8038 leader_election.cc:288] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d735da26ef5446078f2136fbfca8a255 (127.6.209.193:36233), 9cfb1cde0d0741f69ef6788baae94dbc (127.6.209.194:45985)
I20260926 15:15:00.987354 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:00.981059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:52729 (local address 127.6.209.193:36233)
0926 15:15:00.981529 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:00.981534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:00.981581 (+ 47us) server_negotiation.cc:407] Connection header received
0926 15:15:00.981650 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:00.981655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:00.981728 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:00.981795 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:00.982230 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.983307 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.985539 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.986648 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.986798 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:00.986865 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:00.987017 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 15:15:00.987136 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320}
I20260926 15:15:00.987912 8014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:00.980945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:36233 (local address 127.6.209.195:52729)
0926 15:15:00.981327 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:00.981364 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:00.981498 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:00.981851 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:00.981858 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:00.981886 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:00.982108 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:00.982116 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.983416 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.983420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:00.985404 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:00.985416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.986771 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.986774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:00.986886 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:00.986906 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 15:15:00.986970 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 15:15:00.987110 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":38,"spinlock_wait_cycles":16256}
W20260926 15:15:00.987917 8013 tablet.cc:2369] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:15:00.988289 8027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:00.981103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:57395 (local address 127.6.209.194:45985)
0926 15:15:00.981328 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:00.981336 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:00.981894 (+ 558us) server_negotiation.cc:407] Connection header received
0926 15:15:00.982069 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:00.982074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:00.982122 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:00.982216 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:00.982682 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.983676 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.985782 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.986943 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.987087 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:00.987142 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:00.987896 (+ 754us) server_negotiation.cc:299] Negotiation successful
0926 15:15:00.988036 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 15:15:00.989084 8039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:00.981081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:45985 (local address 127.6.209.195:57395)
0926 15:15:00.981790 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:00.981806 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:00.981958 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:00.982249 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:00.982252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:00.982270 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:00.982541 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:00.982548 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.983804 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.983809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:00.985663 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:00.985671 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:00.987133 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:00.987138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:00.987284 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:00.987321 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 15:15:00.988505 (+ 1184us) client_negotiation.cc:240] Negotiation successful
0926 15:15:00.988680 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":45,"spinlock_wait_cycles":19840,"thread_start_us":405,"threads_started":1}
I20260926 15:15:00.988938 7700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71ee2d5703dd4c1f807c103533c438d9" candidate_uuid: "9e349a2983b24063941df6c84bdda4dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d735da26ef5446078f2136fbfca8a255" is_pre_election: true
I20260926 15:15:00.989526 7700 raft_consensus.cc:2454] T 71ee2d5703dd4c1f807c103533c438d9 P d735da26ef5446078f2136fbfca8a255 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e349a2983b24063941df6c84bdda4dd in term 0.
I20260926 15:15:00.990005 7834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71ee2d5703dd4c1f807c103533c438d9" candidate_uuid: "9e349a2983b24063941df6c84bdda4dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cfb1cde0d0741f69ef6788baae94dbc" is_pre_election: true
I20260926 15:15:00.990146 7901 leader_election.cc:302] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e349a2983b24063941df6c84bdda4dd, d735da26ef5446078f2136fbfca8a255; no voters:
I20260926 15:15:00.990476 8038 raft_consensus.cc:2781] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:15:00.990465 7834 raft_consensus.cc:2454] T 71ee2d5703dd4c1f807c103533c438d9 P 9cfb1cde0d0741f69ef6788baae94dbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e349a2983b24063941df6c84bdda4dd in term 0.
I20260926 15:15:00.990579 8038 raft_consensus.cc:482] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:15:00.990679 8038 raft_consensus.cc:3037] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:00.992624 8038 raft_consensus.cc:504] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d735da26ef5446078f2136fbfca8a255" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 36233 } } peers { permanent_uuid: "9cfb1cde0d0741f69ef6788baae94dbc" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 45985 } } peers { permanent_uuid: "9e349a2983b24063941df6c84bdda4dd" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 32773 } }
I20260926 15:15:00.993254 8038 leader_election.cc:288] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd [CANDIDATE]: Term 1 election: Requested vote from peers d735da26ef5446078f2136fbfca8a255 (127.6.209.193:36233), 9cfb1cde0d0741f69ef6788baae94dbc (127.6.209.194:45985)
I20260926 15:15:00.993556 7700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71ee2d5703dd4c1f807c103533c438d9" candidate_uuid: "9e349a2983b24063941df6c84bdda4dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d735da26ef5446078f2136fbfca8a255"
I20260926 15:15:00.993691 7834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71ee2d5703dd4c1f807c103533c438d9" candidate_uuid: "9e349a2983b24063941df6c84bdda4dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cfb1cde0d0741f69ef6788baae94dbc"
I20260926 15:15:00.993759 7700 raft_consensus.cc:3037] T 71ee2d5703dd4c1f807c103533c438d9 P d735da26ef5446078f2136fbfca8a255 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:00.993875 7834 raft_consensus.cc:3037] T 71ee2d5703dd4c1f807c103533c438d9 P 9cfb1cde0d0741f69ef6788baae94dbc [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:00.995643 7700 raft_consensus.cc:2454] T 71ee2d5703dd4c1f807c103533c438d9 P d735da26ef5446078f2136fbfca8a255 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e349a2983b24063941df6c84bdda4dd in term 1.
I20260926 15:15:00.995738 7834 raft_consensus.cc:2454] T 71ee2d5703dd4c1f807c103533c438d9 P 9cfb1cde0d0741f69ef6788baae94dbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e349a2983b24063941df6c84bdda4dd in term 1.
I20260926 15:15:00.995980 7901 leader_election.cc:302] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e349a2983b24063941df6c84bdda4dd, d735da26ef5446078f2136fbfca8a255; no voters:
I20260926 15:15:00.996253 8038 raft_consensus.cc:2781] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:00.996636 8038 raft_consensus.cc:686] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd [term 1 LEADER]: Becoming Leader. State: Replica: 9e349a2983b24063941df6c84bdda4dd, State: Running, Role: LEADER
I20260926 15:15:00.997088 8038 consensus_queue.cc:237] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d735da26ef5446078f2136fbfca8a255" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 36233 } } peers { permanent_uuid: "9cfb1cde0d0741f69ef6788baae94dbc" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 45985 } } peers { permanent_uuid: "9e349a2983b24063941df6c84bdda4dd" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 32773 } }
I20260926 15:15:01.001614 7555 catalog_manager.cc:5495] T 71ee2d5703dd4c1f807c103533c438d9 P 9e349a2983b24063941df6c84bdda4dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e349a2983b24063941df6c84bdda4dd (127.6.209.195). New cstate: current_term: 1 leader_uuid: "9e349a2983b24063941df6c84bdda4dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d735da26ef5446078f2136fbfca8a255" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 36233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cfb1cde0d0741f69ef6788baae94dbc" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 45985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e349a2983b24063941df6c84bdda4dd" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 32773 } health_report { overall_health: HEALTHY } } }
Sep 26 15:15:01 dist-test-slave-j6q7 krb5kdc[7511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.193@KRBTEST.COM
I20260926 15:15:01.026883 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:01.014329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58762 (local address 127.6.209.193:36233)
0926 15:15:01.014483 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:01.014490 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:01.014935 (+ 445us) server_negotiation.cc:407] Connection header received
0926 15:15:01.015047 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:01.015050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:01.015109 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:01.015175 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:01.016351 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:01.017030 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:01.017999 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:01.018336 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:01.018413 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:01.021681 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:01.021691 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:01.021704 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:01.021733 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:01.024520 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:01.025131 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:01.025137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:01.025139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:01.025241 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:01.025580 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:01.025584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:01.025589 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:01.025915 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:01.026191 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:01.026439 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 15:15:01.026622 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 15:15:01.037456 7880 tablet.cc:2369] T 71ee2d5703dd4c1f807c103533c438d9 P 9cfb1cde0d0741f69ef6788baae94dbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 15:15:01 dist-test-slave-j6q7 krb5kdc[7511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435701, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 15:15:01 dist-test-slave-j6q7 krb5kdc[7511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435701, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.209.193@KRBTEST.COM
I20260926 15:15:01.064910 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:01.053516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58768 (local address 127.6.209.193:36233)
0926 15:15:01.053725 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:01.053730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:01.053858 (+ 128us) server_negotiation.cc:407] Connection header received
0926 15:15:01.053970 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:01.053973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:01.054022 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:01.054093 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:01.055123 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:01.055786 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:01.057007 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:01.057375 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:01.057458 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:01.060604 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:01.060608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:01.060611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:01.060634 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:01.062881 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:01.063370 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:01.063373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:01.063375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:01.063490 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:01.063823 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:01.063826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:01.063828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:01.064047 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:01.064197 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:01.064426 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 15:15:01.064547 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 15:15:01 dist-test-slave-j6q7 krb5kdc[7511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 15:15:01 dist-test-slave-j6q7 krb5kdc[7511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.193@KRBTEST.COM
I20260926 15:15:01.101722 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:01.090102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58772 (local address 127.6.209.193:36233)
0926 15:15:01.090307 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:01.090312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:01.090444 (+ 132us) server_negotiation.cc:407] Connection header received
0926 15:15:01.090634 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:01.090638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:01.090691 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:01.090780 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:01.091863 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:01.092457 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:01.093718 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:01.094136 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:01.094247 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:01.097569 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:01.097574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:01.097582 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:01.097607 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:01.099703 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:01.100236 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:01.100239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:01.100241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:01.100347 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:01.100613 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:01.100616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:01.100617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:01.100925 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:01.101087 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:01.101310 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 15:15:01.101467 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 15:15:01.103355 7660 tablet_service.cc:2908] Rejecting scan request for tablet 71ee2d5703dd4c1f807c103533c438d9: Uninitialized: clean time has not yet been initialized
I20260926 15:15:01.106009 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 7617
W20260926 15:15:01.111593 7746 tablet.cc:2369] T 71ee2d5703dd4c1f807c103533c438d9 P d735da26ef5446078f2136fbfca8a255: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:15:01.116470 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:01.106647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.6.209.193:36233)
0926 15:15:01.106805 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:01.106812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:01.107054 (+ 242us) server_negotiation.cc:407] Connection header received
0926 15:15:01.107140 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:01.107143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:01.107197 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:01.107259 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:01.108167 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:01.108735 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:01.109698 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:01.110052 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:01.110138 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:01.111888 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:01.111893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:01.111899 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:01.111929 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:01.114651 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:01.115155 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:01.115158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:01.115160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:01.115246 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:01.115496 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:01.115498 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:01.115500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:01.115707 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:01.115872 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:01.116057 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 15:15:01.116182 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 15:15:01.117038 7740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:01.196563 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 7617
I20260926 15:15:01.212746 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 7751
Sep 26 15:15:01 dist-test-slave-j6q7 krb5kdc[7511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.194@KRBTEST.COM
I20260926 15:15:01.225404 8027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:01.213524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49448 (local address 127.6.209.194:45985)
0926 15:15:01.213702 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:01.213721 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:01.213739 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:15:01.213929 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:01.213933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:01.213999 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:01.214093 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:01.215085 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:01.215720 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:01.216976 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:01.217309 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:01.217430 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:01.220713 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:01.220726 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:01.220739 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:01.220764 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:01.223228 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:01.223796 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:01.223802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:01.223804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:01.223909 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:01.224200 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:01.224202 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:01.224204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:01.224519 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:01.224722 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:01.224939 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 15:15:01.225100 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 15:15:01.225955 7874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:01.303386 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 7751
I20260926 15:15:01.320561 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 7884
Sep 26 15:15:01 dist-test-slave-j6q7 krb5kdc[7511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.195@KRBTEST.COM
I20260926 15:15:01.335201 8029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:01.321324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46074 (local address 127.6.209.195:32773)
0926 15:15:01.321511 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:01.321517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:01.321538 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:15:01.321677 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:01.321682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:01.321768 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:01.321866 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:01.322970 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:01.324041 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:01.325302 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:01.325927 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:01.326039 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:01.329721 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:01.329736 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:01.329755 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:01.329788 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:01.333155 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:01.333721 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:01.333728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:01.333731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:01.333873 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:01.334182 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:01.334185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:01.334186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:01.334461 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:01.334657 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:01.334838 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 15:15:01.334986 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 15:15:01.335773 8007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:01.418496 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 7884
I20260926 15:15:01.433703 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 7524
I20260926 15:15:01.434722 7586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:01.531512 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 7524
2026-09-26T15:15:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3691 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:15:01 dist-test-slave-j6q7 krb5kdc[8059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:15:01 dist-test-slave-j6q7 krb5kdc[8059](info): set up 2 sockets
Sep 26 15:15:01 dist-test-slave-j6q7 krb5kdc[8059](info): commencing operation
krb5kdc: starting...
W20260926 15:15:02.529022 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:15:02 dist-test-slave-j6q7 krb5kdc[8059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:15:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:15:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:15:02.691601 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:40837
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:40837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:02.974649 8072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:02.974934 8072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:02.975092 8072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:02.981070 8072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:15:02.981199 8072 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 15:15:02.981307 8072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:02.981416 8072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:02.981513 8072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:15:02.981596 8072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:15:02.989930 8072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:40837
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:40837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:02.991021 8072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:02.992130 8072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:03.002712 8080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:03.002879 8077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:03.002794 8078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:03.003731 8072 server_base.cc:1034] running on GCE node
I20260926 15:15:03.004509 8072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:03.005716 8072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:03.006974 8072 hybrid_clock.cc:648] HybridClock initialized: now 1790435703006896 us; error 25 us; skew 500 ppm
I20260926 15:15:03.007328 8072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:03 dist-test-slave-j6q7 krb5kdc[8059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435703, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:03.010066 8072 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:15:03.011358 8072 webserver.cc:460] Webserver started at http://127.6.209.254:44507/ using document root <none> and password file <none>
I20260926 15:15:03.011912 8072 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:03.011996 8072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:03.012221 8072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:03.014856 8072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "911f623fb5f545b4972ec63846443d40"
format_stamp: "Formatted at 2026-09-26 15:15:03 on dist-test-slave-j6q7"
server_key: "aea7dd05515ce3efbe8b783c5974e6e7"
server_key_iv: "a93d36049e7d6c94d5d7708e9d998d43"
server_key_version: "encryptionkey@0"
I20260926 15:15:03.015498 8072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "911f623fb5f545b4972ec63846443d40"
format_stamp: "Formatted at 2026-09-26 15:15:03 on dist-test-slave-j6q7"
server_key: "aea7dd05515ce3efbe8b783c5974e6e7"
server_key_iv: "a93d36049e7d6c94d5d7708e9d998d43"
server_key_version: "encryptionkey@0"
I20260926 15:15:03.019692 8072 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 15:15:03.022658 8087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:03.023862 8072 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:15:03.024045 8072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "911f623fb5f545b4972ec63846443d40"
format_stamp: "Formatted at 2026-09-26 15:15:03 on dist-test-slave-j6q7"
server_key: "aea7dd05515ce3efbe8b783c5974e6e7"
server_key_iv: "a93d36049e7d6c94d5d7708e9d998d43"
server_key_version: "encryptionkey@0"
I20260926 15:15:03.024224 8072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:03.044356 8072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:03.047349 8072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:03.047575 8072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:03.066529 8072 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:40837
I20260926 15:15:03.066558 8148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:40837 every 8 connection(s)
I20260926 15:15:03.067593 8072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:15:03.070513 8149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:03.073973 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 8072
I20260926 15:15:03.074157 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:15:03.074714 6983 external_mini_cluster.cc:1428] Setting key 848df72f7b76c9c594a15216735ecccd
I20260926 15:15:03.079777 8149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40: Bootstrap starting.
I20260926 15:15:03.082954 8149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:03.084035 8149 log.cc:824] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:03.086988 8149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40: No bootstrap required, opened a new log
Sep 26 15:15:03 dist-test-slave-j6q7 krb5kdc[8059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:03.093571 8149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911f623fb5f545b4972ec63846443d40" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40837 } }
I20260926 15:15:03.093954 8149 raft_consensus.cc:374] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:03.094049 8149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 911f623fb5f545b4972ec63846443d40, State: Initialized, Role: FOLLOWER
I20260926 15:15:03.094542 8149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911f623fb5f545b4972ec63846443d40" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40837 } }
I20260926 15:15:03.094748 8149 raft_consensus.cc:388] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:15:03.094866 8149 raft_consensus.cc:482] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:15:03.095008 8149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:03.095090 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:03.076318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47736 (local address 127.6.209.254:40837)
0926 15:15:03.077128 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:03.077137 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:03.077177 (+ 40us) server_negotiation.cc:407] Connection header received
0926 15:15:03.078653 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:03.078671 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:03.078973 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:03.079223 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:03.080655 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:03.081903 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:03.083595 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:03.084260 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:03.084366 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:03.088598 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:03.088604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:03.088615 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:03.088646 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:03.091311 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:03.091912 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:03.091922 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:03.091929 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:03.092078 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:03.092410 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:03.092414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:03.092419 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:03.092914 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:03.093199 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:03.093610 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 15:15:03.093989 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":327,"threads_started":1}
I20260926 15:15:03.096937 8149 raft_consensus.cc:504] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911f623fb5f545b4972ec63846443d40" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40837 } }
I20260926 15:15:03.097409 8149 leader_election.cc:302] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 911f623fb5f545b4972ec63846443d40; no voters:
I20260926 15:15:03.097935 8149 leader_election.cc:288] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:15:03.098054 8154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:03.098733 8154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40 [term 1 LEADER]: Becoming Leader. State: Replica: 911f623fb5f545b4972ec63846443d40, State: Running, Role: LEADER
I20260926 15:15:03.099339 8154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911f623fb5f545b4972ec63846443d40" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40837 } }
I20260926 15:15:03.099834 8149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:15:03.101936 8155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "911f623fb5f545b4972ec63846443d40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911f623fb5f545b4972ec63846443d40" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40837 } } }
I20260926 15:15:03.101913 8156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 911f623fb5f545b4972ec63846443d40. Latest consensus state: current_term: 1 leader_uuid: "911f623fb5f545b4972ec63846443d40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911f623fb5f545b4972ec63846443d40" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40837 } } }
I20260926 15:15:03.102326 8155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:03.102326 8156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:03.104279 8159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:15:03.109501 8159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:15:03.116732 8159 catalog_manager.cc:1347] Generated new cluster ID: fc00af36566b459c92efb4ee573a08b6
I20260926 15:15:03.116839 8159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:15:03.131906 8159 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:15:03.133114 8159 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:15:03.139763 8159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40: Generated new TSK 0
I20260926 15:15:03.140349 8159 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
I20260926 15:15:03.178272 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:40837
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:03.462910 8175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:03.463147 8175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:03.463301 8175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:03.469413 8175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:03.469506 8175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:03.469642 8175 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:15:03.469743 8175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:15:03.477753 8175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.209.254:40837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:03.478672 8175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:03.479780 8175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:03.490955 8180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:03.490872 8181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:03.491312 8183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:03.491609 8175 server_base.cc:1034] running on GCE node
I20260926 15:15:03.492130 8175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:03.492959 8175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:03.494189 8175 hybrid_clock.cc:648] HybridClock initialized: now 1790435703494130 us; error 38 us; skew 500 ppm
I20260926 15:15:03.494438 8175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:03 dist-test-slave-j6q7 krb5kdc[8059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435703, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:03.496986 8175 init.cc:377] Logged in from keytab as kudu/127.6.209.193@KRBTEST.COM (short username kudu)
I20260926 15:15:03.498195 8175 webserver.cc:460] Webserver started at http://127.6.209.193:39881/ using document root <none> and password file <none>
I20260926 15:15:03.498785 8175 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:03.498903 8175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:03.499173 8175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:03.501878 8175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "e776fe34da26474480744b8040065764"
format_stamp: "Formatted at 2026-09-26 15:15:03 on dist-test-slave-j6q7"
server_key: "b3bae1eb58f6fbd64ed1bcd7ae5e8546"
server_key_iv: "8e9bef9ccce7fb472721d821373f1928"
server_key_version: "encryptionkey@0"
I20260926 15:15:03.502600 8175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "e776fe34da26474480744b8040065764"
format_stamp: "Formatted at 2026-09-26 15:15:03 on dist-test-slave-j6q7"
server_key: "b3bae1eb58f6fbd64ed1bcd7ae5e8546"
server_key_iv: "8e9bef9ccce7fb472721d821373f1928"
server_key_version: "encryptionkey@0"
I20260926 15:15:03.506826 8175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 15:15:03.509711 8190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:03.510740 8175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:15:03.510891 8175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "e776fe34da26474480744b8040065764"
format_stamp: "Formatted at 2026-09-26 15:15:03 on dist-test-slave-j6q7"
server_key: "b3bae1eb58f6fbd64ed1bcd7ae5e8546"
server_key_iv: "8e9bef9ccce7fb472721d821373f1928"
server_key_version: "encryptionkey@0"
I20260926 15:15:03.511103 8175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:03.524410 8175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:03.528002 8175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:03.528232 8175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:03.530400 8175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:03.530488 8175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:03.530625 8175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:03.530706 8175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 15:15:03 dist-test-slave-j6q7 krb5kdc[8059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435703, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:03.551349 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:03.530941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:54559 (local address 127.6.209.254:40837)
0926 15:15:03.531133 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:03.531140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:03.532286 (+ 1146us) server_negotiation.cc:407] Connection header received
0926 15:15:03.534226 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:03.534231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:03.534324 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:03.534419 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:03.536356 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:03.537643 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:03.539886 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:03.540678 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:03.540853 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:03.545023 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:03.545031 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:03.545036 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:03.545088 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:03.548221 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:03.549004 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:03.549011 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:03.549015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:03.549194 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:03.549556 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:03.549563 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:03.549567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:03.549935 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:03.550088 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:03.550671 (+ 583us) server_negotiation.cc:299] Negotiation successful
0926 15:15:03.550921 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 15:15:03.551661 8199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:03.531435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40837 (local address 127.6.209.193:54559)
0926 15:15:03.532144 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:03.532180 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:03.533962 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:03.534637 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:03.534647 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:03.535310 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:03.536146 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:03.536173 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:03.537815 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:03.537821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:03.539698 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:03.539710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:03.540919 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:03.540924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:03.541088 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:03.541735 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:03.541769 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:03.544803 (+ 3034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:03.548354 (+ 3551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:03.548365 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:03.548381 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:03.548891 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:03.549300 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:03.549303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:03.549305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:03.549457 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:03.550054 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:03.550058 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:03.550352 (+ 294us) client_negotiation.cc:769] Sending connection context
0926 15:15:03.550523 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 15:15:03.550790 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":326,"threads_started":1}
I20260926 15:15:03.570298 8175 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:39569
I20260926 15:15:03.570281 8309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:39569 every 8 connection(s)
I20260926 15:15:03.571339 8175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:15:03.573202 8310 heartbeater.cc:347] Connected to a master server at 127.6.209.254:40837
I20260926 15:15:03.573415 8310 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:03.574110 8310 heartbeater.cc:511] Master 127.6.209.254:40837 requested a full tablet report, sending...
I20260926 15:15:03.576030 8104 ts_manager.cc:194] Registered new tserver with Master: e776fe34da26474480744b8040065764 (127.6.209.193:39569)
I20260926 15:15:03.577535 8104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.193@KRBTEST.COM'} at 127.6.209.193:54559
I20260926 15:15:03.580101 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:03.567270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:59357 (local address 127.6.209.254:40837)
0926 15:15:03.567440 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:03.567447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:03.569488 (+ 2041us) server_negotiation.cc:407] Connection header received
0926 15:15:03.569714 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:03.569720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:03.569771 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:03.569855 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:03.570740 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:03.571385 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:03.573335 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:03.573765 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:03.573854 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:03.575466 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:03.575471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:03.575478 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:03.575509 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:03.578009 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:03.578637 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:03.578640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:03.578642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:03.578746 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:03.579096 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:03.579099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:03.579101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:03.579327 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:03.579429 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:03.579781 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 15:15:03.579902 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 15:15:03.580328 8301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:03.566866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40837 (local address 127.6.209.193:59357)
0926 15:15:03.569390 (+ 2524us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:03.569410 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:03.569593 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:03.569881 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:03.569885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:03.570298 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:03.570629 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:03.570637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:03.571508 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:03.571513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:03.573190 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:03.573205 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:03.573957 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:03.573963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:03.574148 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:03.574587 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:03.574608 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:03.575326 (+ 718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:03.578125 (+ 2799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:03.578133 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:03.578137 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:03.578536 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:03.578855 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:03.578860 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:03.578863 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:03.579005 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:03.579426 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:03.579430 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:03.579587 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 15:15:03.579716 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 15:15:03.579933 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2392,"thread_start_us":2268,"threads_started":1}
I20260926 15:15:03.580901 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 8175
I20260926 15:15:03.581097 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:15:03.581708 6983 external_mini_cluster.cc:1428] Setting key 9990cbc172dcd1fc64fb96fd8474af6c
WARNING: no policy specified for kudu/127.6.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
I20260926 15:15:03.616170 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:40837
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:03.898645 8314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:03.898860 8314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:03.898947 8314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:03.905478 8314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:03.905566 8314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:03.905644 8314 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:15:03.905712 8314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:15:03.914052 8314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.209.254:40837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:03.915020 8314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:03.916024 8314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:03.927560 8319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:03.927627 8320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:03.927944 8322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:03.928126 8314 server_base.cc:1034] running on GCE node
I20260926 15:15:03.928792 8314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:03.929582 8314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:03.930796 8314 hybrid_clock.cc:648] HybridClock initialized: now 1790435703930748 us; error 47 us; skew 500 ppm
I20260926 15:15:03.931034 8314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:03 dist-test-slave-j6q7 krb5kdc[8059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435703, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:03.933710 8314 init.cc:377] Logged in from keytab as kudu/127.6.209.194@KRBTEST.COM (short username kudu)
I20260926 15:15:03.935038 8314 webserver.cc:460] Webserver started at http://127.6.209.194:39735/ using document root <none> and password file <none>
I20260926 15:15:03.935654 8314 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:03.935774 8314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:03.936056 8314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:03.938612 8314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "d976da13ccef419ea690cc6682cafebb"
format_stamp: "Formatted at 2026-09-26 15:15:03 on dist-test-slave-j6q7"
server_key: "ca412b81337c4e1b35f0c54a8efe9736"
server_key_iv: "7b44f0961842b9789818695a4888eb51"
server_key_version: "encryptionkey@0"
I20260926 15:15:03.939262 8314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "d976da13ccef419ea690cc6682cafebb"
format_stamp: "Formatted at 2026-09-26 15:15:03 on dist-test-slave-j6q7"
server_key: "ca412b81337c4e1b35f0c54a8efe9736"
server_key_iv: "7b44f0961842b9789818695a4888eb51"
server_key_version: "encryptionkey@0"
I20260926 15:15:03.943622 8314 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 15:15:03.946887 8329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:03.948004 8314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:15:03.948128 8314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "d976da13ccef419ea690cc6682cafebb"
format_stamp: "Formatted at 2026-09-26 15:15:03 on dist-test-slave-j6q7"
server_key: "ca412b81337c4e1b35f0c54a8efe9736"
server_key_iv: "7b44f0961842b9789818695a4888eb51"
server_key_version: "encryptionkey@0"
I20260926 15:15:03.948264 8314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:03.975041 8314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:03.978147 8314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:03.978318 8314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:03.980361 8314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:03.980484 8314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:03.980616 8314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:03.980683 8314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 15:15:03 dist-test-slave-j6q7 krb5kdc[8059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435703, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:04.001735 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:03.981089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:46753 (local address 127.6.209.254:40837)
0926 15:15:03.981285 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:03.981292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:03.982491 (+ 1199us) server_negotiation.cc:407] Connection header received
0926 15:15:03.984292 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:03.984299 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:03.984375 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:03.984489 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:03.986436 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:03.987665 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:03.990048 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:03.990628 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:03.990756 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:03.995110 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:03.995115 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:03.995117 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:03.995144 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:03.997763 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:03.998544 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:03.998549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:03.998552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:03.998666 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:03.999202 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:03.999210 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:03.999216 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:03.999593 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:03.999752 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:04.001043 (+ 1291us) server_negotiation.cc:299] Negotiation successful
0926 15:15:04.001322 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 15:15:04.003206 8338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:03.981635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40837 (local address 127.6.209.194:46753)
0926 15:15:03.982362 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:03.982391 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:03.984035 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:03.984683 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:03.984689 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:03.985347 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:03.986243 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:03.986270 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:03.987829 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:03.987835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:03.989919 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:03.989931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:03.990859 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:03.990866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:03.991094 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:03.991802 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:03.991821 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:03.994900 (+ 3079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:03.997889 (+ 2989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:03.997900 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:03.997916 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:03.998412 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:03.998783 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:03.998787 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:03.998791 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:03.999057 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:03.999828 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:03.999836 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:04.000345 (+ 509us) client_negotiation.cc:769] Sending connection context
0926 15:15:04.000723 (+ 378us) client_negotiation.cc:240] Negotiation successful
0926 15:15:04.001344 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":310,"threads_started":1}
I20260926 15:15:04.023003 8314 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:36291
I20260926 15:15:04.023057 8448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:36291 every 8 connection(s)
I20260926 15:15:04.024175 8314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:15:04.026098 8449 heartbeater.cc:347] Connected to a master server at 127.6.209.254:40837
I20260926 15:15:04.026338 8449 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:04.027005 8449 heartbeater.cc:511] Master 127.6.209.254:40837 requested a full tablet report, sending...
I20260926 15:15:04.028219 8104 ts_manager.cc:194] Registered new tserver with Master: d976da13ccef419ea690cc6682cafebb (127.6.209.194:36291)
I20260926 15:15:04.028721 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 8314
I20260926 15:15:04.028929 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:15:04.029177 8104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.194@KRBTEST.COM'} at 127.6.209.194:46753
I20260926 15:15:04.029843 6983 external_mini_cluster.cc:1428] Setting key e06b01ab195664311fdaef60a4d4bd1c
I20260926 15:15:04.033134 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:04.019570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:49079 (local address 127.6.209.254:40837)
0926 15:15:04.019726 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:04.019733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:04.021364 (+ 1631us) server_negotiation.cc:407] Connection header received
0926 15:15:04.021534 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:04.021539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:04.021605 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:04.021698 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:04.022781 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.023658 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.025054 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.025664 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.025759 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:04.027535 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:04.027540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:04.027545 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:04.027580 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:04.030773 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:04.031481 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:04.031485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:04.031488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:04.031615 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:04.031948 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:04.031951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:04.031952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:04.032189 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:04.032282 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:04.032692 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 15:15:04.032893 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 15:15:04.033530 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:04.019749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40837 (local address 127.6.209.194:49079)
0926 15:15:04.021274 (+ 1525us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:04.021289 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:04.021418 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:04.021722 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:04.021726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:04.022334 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:04.022652 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:04.022660 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.023799 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.023804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:04.024862 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:04.024878 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.026002 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.026006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:04.026124 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:04.026722 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:04.026735 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:04.027380 (+ 645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:04.030902 (+ 3522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:04.030907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:04.030910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:04.031384 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:04.031713 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:04.031719 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:04.031722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:04.031854 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:04.032299 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:04.032306 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:04.032469 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 15:15:04.032627 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 15:15:04.032979 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1373,"thread_start_us":1231,"threads_started":1}
WARNING: no policy specified for kudu/127.6.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
I20260926 15:15:04.065774 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:40837
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:04.364699 8453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:04.364964 8453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:04.365118 8453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:04.371666 8453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:04.371755 8453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:04.371912 8453 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:15:04.372031 8453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:15:04.380791 8453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.209.254:40837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:04.381924 8453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:04.383113 8453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:04.394825 8458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:04.394907 8459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:04.394907 8461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:04.395888 8453 server_base.cc:1034] running on GCE node
I20260926 15:15:04.396543 8453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:04.397375 8453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:04.398572 8453 hybrid_clock.cc:648] HybridClock initialized: now 1790435704398524 us; error 25 us; skew 500 ppm
I20260926 15:15:04.398847 8453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:04 dist-test-slave-j6q7 krb5kdc[8059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435704, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:04.401746 8453 init.cc:377] Logged in from keytab as kudu/127.6.209.195@KRBTEST.COM (short username kudu)
I20260926 15:15:04.403076 8453 webserver.cc:460] Webserver started at http://127.6.209.195:41685/ using document root <none> and password file <none>
I20260926 15:15:04.403697 8453 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:04.403806 8453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:04.404122 8453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:04.406783 8453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "367b737530864214bdf43d409f5c7455"
format_stamp: "Formatted at 2026-09-26 15:15:04 on dist-test-slave-j6q7"
server_key: "5f66cb81c3a2e8b686f63af5a695fdd4"
server_key_iv: "3f5603c6ba7c73e0f46a6ee0ff3dc61f"
server_key_version: "encryptionkey@0"
I20260926 15:15:04.407523 8453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "367b737530864214bdf43d409f5c7455"
format_stamp: "Formatted at 2026-09-26 15:15:04 on dist-test-slave-j6q7"
server_key: "5f66cb81c3a2e8b686f63af5a695fdd4"
server_key_iv: "3f5603c6ba7c73e0f46a6ee0ff3dc61f"
server_key_version: "encryptionkey@0"
I20260926 15:15:04.412130 8453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 15:15:04.415382 8468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:04.416661 8453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:15:04.416893 8453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "367b737530864214bdf43d409f5c7455"
format_stamp: "Formatted at 2026-09-26 15:15:04 on dist-test-slave-j6q7"
server_key: "5f66cb81c3a2e8b686f63af5a695fdd4"
server_key_iv: "3f5603c6ba7c73e0f46a6ee0ff3dc61f"
server_key_version: "encryptionkey@0"
I20260926 15:15:04.417112 8453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:04.432539 8453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:04.436050 8453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:04.436291 8453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:04.438436 8453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:04.438593 8453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:04.438737 8453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:04.438809 8453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 15:15:04 dist-test-slave-j6q7 krb5kdc[8059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435704, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:04.459725 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:04.439018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:42909 (local address 127.6.209.254:40837)
0926 15:15:04.439259 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:04.439265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:04.440481 (+ 1216us) server_negotiation.cc:407] Connection header received
0926 15:15:04.442480 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:04.442488 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:04.442561 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:04.442674 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:04.444574 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.445751 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.448071 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.448585 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.448709 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:04.453620 (+ 4911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:04.453628 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:04.453632 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:04.453672 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:04.456907 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:04.457695 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:04.457700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:04.457703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:04.457857 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:04.458319 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:04.458325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:04.458330 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:04.458551 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:04.458681 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:04.459225 (+ 544us) server_negotiation.cc:299] Negotiation successful
0926 15:15:04.459412 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260926 15:15:04.460451 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:04.439566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40837 (local address 127.6.209.195:42909)
0926 15:15:04.440314 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:04.440353 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:04.442155 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:04.442898 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:04.442905 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:04.443628 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:04.444370 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:04.444386 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.445918 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.445923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:04.447885 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:04.447903 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.448791 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.448797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:04.449167 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:04.449833 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:04.449866 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:04.453363 (+ 3497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:04.457065 (+ 3702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:04.457075 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:04.457091 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:04.457565 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:04.457983 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:04.457995 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:04.457999 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:04.458209 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:04.458659 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:04.458666 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:04.458932 (+ 266us) client_negotiation.cc:769] Sending connection context
0926 15:15:04.459143 (+ 211us) client_negotiation.cc:240] Negotiation successful
0926 15:15:04.459507 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":332,"threads_started":1}
I20260926 15:15:04.482149 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:04.466347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:60021 (local address 127.6.209.254:40837)
0926 15:15:04.466547 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:04.466552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:04.469548 (+ 2996us) server_negotiation.cc:407] Connection header received
0926 15:15:04.469661 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:04.469668 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:04.469751 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:04.469861 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:04.470579 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.472035 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.473447 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.474227 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.474376 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:04.476483 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:04.476491 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:04.476495 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:04.476532 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:04.479428 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:04.480286 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:04.480293 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:04.480297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:04.480431 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:04.480746 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:04.480750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:04.480753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:04.481034 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:04.481199 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:04.481588 (+ 389us) server_negotiation.cc:299] Negotiation successful
0926 15:15:04.481779 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 15:15:04.482244 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:04.466279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40837 (local address 127.6.209.195:60021)
0926 15:15:04.469458 (+ 3179us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:04.469472 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:04.469560 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:04.469873 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:04.469876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:04.470235 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:04.470465 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:04.470472 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.472164 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.472168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:04.473297 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:04.473312 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.474435 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.474443 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:04.474638 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:04.475194 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:04.475222 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:04.476256 (+ 1034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:04.479555 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:04.479562 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:04.479573 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:04.480178 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:04.480529 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:04.480534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:04.480537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:04.480645 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:04.481156 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:04.481160 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:04.481348 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 15:15:04.481482 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 15:15:04.481781 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3059,"thread_start_us":2825,"threads_started":1}
I20260926 15:15:04.489241 8453 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:33525
I20260926 15:15:04.489305 8587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:33525 every 8 connection(s)
I20260926 15:15:04.490422 8453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:15:04.492136 8588 heartbeater.cc:347] Connected to a master server at 127.6.209.254:40837
I20260926 15:15:04.492288 8588 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:04.492775 8588 heartbeater.cc:511] Master 127.6.209.254:40837 requested a full tablet report, sending...
I20260926 15:15:04.493975 8104 ts_manager.cc:194] Registered new tserver with Master: 367b737530864214bdf43d409f5c7455 (127.6.209.195:33525)
I20260926 15:15:04.494657 8104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.195@KRBTEST.COM'} at 127.6.209.195:42909
I20260926 15:15:04.500308 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 8453
I20260926 15:15:04.500485 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:15:04.500902 6983 external_mini_cluster.cc:1428] Setting key 754ce1abe988c29cacdc10df8cbfd7fe
I20260926 15:15:04.503322 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 15:15:04 dist-test-slave-j6q7 krb5kdc[8059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435704, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 15:15:04 dist-test-slave-j6q7 krb5kdc[8059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435704, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:04.543473 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:04.531608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47750 (local address 127.6.209.254:40837)
0926 15:15:04.531834 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:04.531839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:04.532035 (+ 196us) server_negotiation.cc:407] Connection header received
0926 15:15:04.532170 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:04.532174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:04.532231 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:04.532300 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:04.533544 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.534287 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.535406 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.535750 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.535828 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:04.539314 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:04.539319 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:04.539322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:04.539352 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:04.541552 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:04.542065 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:04.542068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:04.542070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:04.542176 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:04.542475 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:04.542478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:04.542479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:04.542689 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:04.542847 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:04.543084 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 15:15:04.543247 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 15:15:04.546947 8104 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:15:04.553300 8104 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:15:04.573000 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:04.563807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:33525 (local address 127.0.0.1:56574)
0926 15:15:04.564549 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:04.564591 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:04.564765 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:04.566041 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:04.566059 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:04.566099 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:04.566459 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:04.566468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.568320 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.568327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:04.570346 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:04.570358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.572130 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.572136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:04.572306 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:04.572357 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 15:15:04.572450 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 15:15:04.572616 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":327,"threads_started":1}
I20260926 15:15:04.573912 8603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:04.564206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:39569 (local address 127.0.0.1:41510)
0926 15:15:04.566009 (+ 1803us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:04.566023 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:04.566125 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:04.566927 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:04.566933 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:04.566958 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:04.567344 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:04.567354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.568887 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.568897 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:04.570962 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:04.570974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.573075 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.573081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:04.573273 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:04.573352 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 15:15:04.573442 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 15:15:04.573551 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1674,"mutex_wait_us":101,"thread_start_us":1556,"threads_started":1}
I20260926 15:15:04.574122 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:04.564058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.6.209.195:33525)
0926 15:15:04.565423 (+ 1365us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:04.565431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:04.565461 (+ 30us) server_negotiation.cc:407] Connection header received
0926 15:15:04.565549 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:04.565556 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:04.565797 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:04.566025 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:04.566680 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.568199 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.570486 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.571920 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.573417 (+ 1497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:04.573646 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:04.573722 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 15:15:04.573825 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1203,"thread_start_us":1037,"threads_started":1}
I20260926 15:15:04.574630 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:04.564656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.6.209.193:39569)
0926 15:15:04.565626 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:04.565633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:04.566243 (+ 610us) server_negotiation.cc:407] Connection header received
0926 15:15:04.566342 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:04.566351 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:04.566570 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:04.566745 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:04.567495 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.568751 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.571144 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.572533 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.572670 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:04.572902 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:04.574230 (+ 1328us) server_negotiation.cc:299] Negotiation successful
0926 15:15:04.574371 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":838,"thread_start_us":736,"threads_started":1}
I20260926 15:15:04.574846 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:04.564917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:36291 (local address 127.0.0.1:45242)
0926 15:15:04.566960 (+ 2043us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:04.566975 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:04.567097 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:04.567693 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:04.567698 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:04.567720 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:04.568018 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:04.568026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.569391 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.569396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:04.571090 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:04.571103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.573011 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.573021 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:04.573218 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:04.574359 (+ 1141us) client_negotiation.cc:769] Sending connection context
0926 15:15:04.574444 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 15:15:04.574548 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1916,"mutex_wait_us":1075,"thread_start_us":1795,"threads_started":1}
I20260926 15:15:04.575482 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:04.565339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45242 (local address 127.6.209.194:36291)
0926 15:15:04.566576 (+ 1237us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:04.566584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:04.567209 (+ 625us) server_negotiation.cc:407] Connection header received
0926 15:15:04.567312 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:04.567319 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:04.567500 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:04.567693 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:04.568192 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.569267 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.571260 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.572520 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.572645 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:04.573269 (+ 624us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:04.575104 (+ 1835us) server_negotiation.cc:299] Negotiation successful
0926 15:15:04.575250 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1112,"thread_start_us":1000,"threads_started":1}
I20260926 15:15:04.577554 8518 tablet_service.cc:1467] Processing CreateTablet for tablet 978a090ca2fb440ca27c9c642c443710 (DEFAULT_TABLE table=test-table [id=40f08a29a9874617b788487b926923ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:04.577934 8240 tablet_service.cc:1467] Processing CreateTablet for tablet 978a090ca2fb440ca27c9c642c443710 (DEFAULT_TABLE table=test-table [id=40f08a29a9874617b788487b926923ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:04.578892 8518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 978a090ca2fb440ca27c9c642c443710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:04.579041 8240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 978a090ca2fb440ca27c9c642c443710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:04.580220 8310 heartbeater.cc:503] Master 127.6.209.254:40837 was elected leader, sending a full tablet report...
I20260926 15:15:04.583137 8379 tablet_service.cc:1467] Processing CreateTablet for tablet 978a090ca2fb440ca27c9c642c443710 (DEFAULT_TABLE table=test-table [id=40f08a29a9874617b788487b926923ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:04.584172 8379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 978a090ca2fb440ca27c9c642c443710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:04.592603 8608 tablet_bootstrap.cc:492] T 978a090ca2fb440ca27c9c642c443710 P 367b737530864214bdf43d409f5c7455: Bootstrap starting.
I20260926 15:15:04.593293 8607 tablet_bootstrap.cc:492] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764: Bootstrap starting.
I20260926 15:15:04.596738 8609 tablet_bootstrap.cc:492] T 978a090ca2fb440ca27c9c642c443710 P d976da13ccef419ea690cc6682cafebb: Bootstrap starting.
I20260926 15:15:04.597170 8607 tablet_bootstrap.cc:654] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:04.597396 8608 tablet_bootstrap.cc:654] T 978a090ca2fb440ca27c9c642c443710 P 367b737530864214bdf43d409f5c7455: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:04.598310 8608 log.cc:824] T 978a090ca2fb440ca27c9c642c443710 P 367b737530864214bdf43d409f5c7455: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:04.598326 8607 log.cc:824] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:04.599728 8609 tablet_bootstrap.cc:654] T 978a090ca2fb440ca27c9c642c443710 P d976da13ccef419ea690cc6682cafebb: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:04.600783 8609 log.cc:824] T 978a090ca2fb440ca27c9c642c443710 P d976da13ccef419ea690cc6682cafebb: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:04.601325 8607 tablet_bootstrap.cc:492] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764: No bootstrap required, opened a new log
I20260926 15:15:04.601325 8608 tablet_bootstrap.cc:492] T 978a090ca2fb440ca27c9c642c443710 P 367b737530864214bdf43d409f5c7455: No bootstrap required, opened a new log
I20260926 15:15:04.601590 8607 ts_tablet_manager.cc:1397] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.005s
I20260926 15:15:04.602221 8608 ts_tablet_manager.cc:1397] T 978a090ca2fb440ca27c9c642c443710 P 367b737530864214bdf43d409f5c7455: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 15:15:04.603385 8609 tablet_bootstrap.cc:492] T 978a090ca2fb440ca27c9c642c443710 P d976da13ccef419ea690cc6682cafebb: No bootstrap required, opened a new log
I20260926 15:15:04.603628 8609 ts_tablet_manager.cc:1397] T 978a090ca2fb440ca27c9c642c443710 P d976da13ccef419ea690cc6682cafebb: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260926 15:15:04.610970 8609 raft_consensus.cc:348] T 978a090ca2fb440ca27c9c642c443710 P d976da13ccef419ea690cc6682cafebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } }
I20260926 15:15:04.611294 8609 raft_consensus.cc:374] T 978a090ca2fb440ca27c9c642c443710 P d976da13ccef419ea690cc6682cafebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:04.611411 8609 raft_consensus.cc:729] T 978a090ca2fb440ca27c9c642c443710 P d976da13ccef419ea690cc6682cafebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d976da13ccef419ea690cc6682cafebb, State: Initialized, Role: FOLLOWER
I20260926 15:15:04.611980 8609 consensus_queue.cc:260] T 978a090ca2fb440ca27c9c642c443710 P d976da13ccef419ea690cc6682cafebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } }
I20260926 15:15:04.612118 8608 raft_consensus.cc:348] T 978a090ca2fb440ca27c9c642c443710 P 367b737530864214bdf43d409f5c7455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } }
I20260926 15:15:04.612579 8608 raft_consensus.cc:374] T 978a090ca2fb440ca27c9c642c443710 P 367b737530864214bdf43d409f5c7455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:04.612725 8608 raft_consensus.cc:729] T 978a090ca2fb440ca27c9c642c443710 P 367b737530864214bdf43d409f5c7455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 367b737530864214bdf43d409f5c7455, State: Initialized, Role: FOLLOWER
I20260926 15:15:04.612636 8607 raft_consensus.cc:348] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } }
I20260926 15:15:04.612965 8449 heartbeater.cc:503] Master 127.6.209.254:40837 was elected leader, sending a full tablet report...
I20260926 15:15:04.613148 8607 raft_consensus.cc:374] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:04.613294 8607 raft_consensus.cc:729] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e776fe34da26474480744b8040065764, State: Initialized, Role: FOLLOWER
I20260926 15:15:04.613569 8609 ts_tablet_manager.cc:1428] T 978a090ca2fb440ca27c9c642c443710 P d976da13ccef419ea690cc6682cafebb: Time spent starting tablet: real 0.010s user 0.006s sys 0.005s
I20260926 15:15:04.613518 8608 consensus_queue.cc:260] T 978a090ca2fb440ca27c9c642c443710 P 367b737530864214bdf43d409f5c7455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } }
I20260926 15:15:04.614056 8607 consensus_queue.cc:260] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } }
I20260926 15:15:04.615299 8608 ts_tablet_manager.cc:1428] T 978a090ca2fb440ca27c9c642c443710 P 367b737530864214bdf43d409f5c7455: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260926 15:15:04.615798 8607 ts_tablet_manager.cc:1428] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260926 15:15:04.616573 8588 heartbeater.cc:503] Master 127.6.209.254:40837 was elected leader, sending a full tablet report...
I20260926 15:15:04.651060 8615 raft_consensus.cc:482] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:15:04.651286 8615 raft_consensus.cc:504] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } }
I20260926 15:15:04.653108 8615 leader_election.cc:288] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 367b737530864214bdf43d409f5c7455 (127.6.209.195:33525), d976da13ccef419ea690cc6682cafebb (127.6.209.194:36291)
I20260926 15:15:04.659742 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:04.653536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:48421 (local address 127.6.209.195:33525)
0926 15:15:04.653758 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:04.653763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:04.654310 (+ 547us) server_negotiation.cc:407] Connection header received
0926 15:15:04.654506 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:04.654513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:04.654590 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:04.654674 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:04.655478 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.656328 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.658320 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.658959 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.659126 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:04.659189 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:04.659428 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 15:15:04.659529 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 15:15:04.659873 8616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:04.653517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:33525 (local address 127.6.209.193:48421)
0926 15:15:04.654212 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:04.654237 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:04.654359 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:04.654918 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:04.654922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:04.654948 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:04.655323 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:04.655334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.656435 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.656440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:04.658194 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:04.658203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.659129 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.659134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:04.659285 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:04.659308 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 15:15:04.659394 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 15:15:04.659495 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":1,"thread_start_us":351,"threads_started":1}
I20260926 15:15:04.659901 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:04.654600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:50871 (local address 127.6.209.194:36291)
0926 15:15:04.654842 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:04.654849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:04.655196 (+ 347us) server_negotiation.cc:407] Connection header received
0926 15:15:04.655303 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:04.655309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:04.655373 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:04.655452 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:04.655949 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.656776 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.658497 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.659122 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.659278 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:04.659341 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:04.659573 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 15:15:04.659676 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 15:15:04.660728 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:04.654347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:36291 (local address 127.6.209.193:50871)
0926 15:15:04.655108 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:04.655119 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:04.655211 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:04.655474 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:04.655478 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:04.655497 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:04.655794 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:04.655803 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.656926 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.656930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:04.658372 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:04.658381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.659322 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.659327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:04.659437 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:04.659456 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 15:15:04.659550 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 15:15:04.659653 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":18,"thread_start_us":521,"threads_started":1}
I20260926 15:15:04.660727 8538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "978a090ca2fb440ca27c9c642c443710" candidate_uuid: "e776fe34da26474480744b8040065764" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "367b737530864214bdf43d409f5c7455" is_pre_election: true
I20260926 15:15:04.661321 8538 raft_consensus.cc:2454] T 978a090ca2fb440ca27c9c642c443710 P 367b737530864214bdf43d409f5c7455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e776fe34da26474480744b8040065764 in term 0.
I20260926 15:15:04.661827 8399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "978a090ca2fb440ca27c9c642c443710" candidate_uuid: "e776fe34da26474480744b8040065764" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d976da13ccef419ea690cc6682cafebb" is_pre_election: true
I20260926 15:15:04.662082 8192 leader_election.cc:302] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 367b737530864214bdf43d409f5c7455, e776fe34da26474480744b8040065764; no voters:
I20260926 15:15:04.662330 8399 raft_consensus.cc:2454] T 978a090ca2fb440ca27c9c642c443710 P d976da13ccef419ea690cc6682cafebb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e776fe34da26474480744b8040065764 in term 0.
I20260926 15:15:04.662483 8615 raft_consensus.cc:2781] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:15:04.662631 8615 raft_consensus.cc:482] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:15:04.662768 8615 raft_consensus.cc:3037] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:04.664960 8615 raft_consensus.cc:504] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } }
I20260926 15:15:04.665453 8615 leader_election.cc:288] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764 [CANDIDATE]: Term 1 election: Requested vote from peers 367b737530864214bdf43d409f5c7455 (127.6.209.195:33525), d976da13ccef419ea690cc6682cafebb (127.6.209.194:36291)
I20260926 15:15:04.665858 8538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "978a090ca2fb440ca27c9c642c443710" candidate_uuid: "e776fe34da26474480744b8040065764" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "367b737530864214bdf43d409f5c7455"
I20260926 15:15:04.665900 8399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "978a090ca2fb440ca27c9c642c443710" candidate_uuid: "e776fe34da26474480744b8040065764" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d976da13ccef419ea690cc6682cafebb"
I20260926 15:15:04.666016 8538 raft_consensus.cc:3037] T 978a090ca2fb440ca27c9c642c443710 P 367b737530864214bdf43d409f5c7455 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:04.666121 8399 raft_consensus.cc:3037] T 978a090ca2fb440ca27c9c642c443710 P d976da13ccef419ea690cc6682cafebb [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:04.668087 8399 raft_consensus.cc:2454] T 978a090ca2fb440ca27c9c642c443710 P d976da13ccef419ea690cc6682cafebb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e776fe34da26474480744b8040065764 in term 1.
I20260926 15:15:04.668099 8538 raft_consensus.cc:2454] T 978a090ca2fb440ca27c9c642c443710 P 367b737530864214bdf43d409f5c7455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e776fe34da26474480744b8040065764 in term 1.
I20260926 15:15:04.668393 8192 leader_election.cc:302] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d976da13ccef419ea690cc6682cafebb, e776fe34da26474480744b8040065764; no voters:
I20260926 15:15:04.668629 8615 raft_consensus.cc:2781] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:04.669075 8615 raft_consensus.cc:686] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764 [term 1 LEADER]: Becoming Leader. State: Replica: e776fe34da26474480744b8040065764, State: Running, Role: LEADER
I20260926 15:15:04.669495 8615 consensus_queue.cc:237] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } }
I20260926 15:15:04.673347 8101 catalog_manager.cc:5495] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764 reported cstate change: term changed from 0 to 1, leader changed from <none> to e776fe34da26474480744b8040065764 (127.6.209.193). New cstate: current_term: 1 leader_uuid: "e776fe34da26474480744b8040065764" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } health_report { overall_health: HEALTHY } } }
I20260926 15:15:04.690970 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:04.686789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41518 (local address 127.6.209.193:39569)
0926 15:15:04.686925 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:04.686928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:04.687081 (+ 153us) server_negotiation.cc:407] Connection header received
0926 15:15:04.687278 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:04.687281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:04.687340 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:04.687433 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:15:04.687876 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.688435 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.689522 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.689845 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.689919 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:04.690252 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:15:04.690608 (+ 356us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:15:04.690692 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:04.690779 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 15:15:04.690825 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 15:15:04.697427 8399 raft_consensus.cc:1261] T 978a090ca2fb440ca27c9c642c443710 P d976da13ccef419ea690cc6682cafebb [term 1 FOLLOWER]: Refusing update from remote peer e776fe34da26474480744b8040065764: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:04.697510 8538 raft_consensus.cc:1261] T 978a090ca2fb440ca27c9c642c443710 P 367b737530864214bdf43d409f5c7455 [term 1 FOLLOWER]: Refusing update from remote peer e776fe34da26474480744b8040065764: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:04.698153 8615 consensus_queue.cc:1035] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:04.698374 8618 consensus_queue.cc:1035] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764 [LEADER]: Connected to new peer: Peer: permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:04.723950 8101 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:04.724633 8101 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:04.727741 8101 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40: Sending DeleteTablet for 3 replicas of tablet 978a090ca2fb440ca27c9c642c443710
I20260926 15:15:04.728853 8379 tablet_service.cc:1514] Processing DeleteTablet for tablet 978a090ca2fb440ca27c9c642c443710 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:04 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:45242
I20260926 15:15:04.728868 8518 tablet_service.cc:1514] Processing DeleteTablet for tablet 978a090ca2fb440ca27c9c642c443710 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:04 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:56574
I20260926 15:15:04.729472 8240 tablet_service.cc:1514] Processing DeleteTablet for tablet 978a090ca2fb440ca27c9c642c443710 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:04 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:41510
I20260926 15:15:04.729923 8101 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:15:04.730934 8101 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:15:04.738006 8239 tablet_service.cc:1467] Processing CreateTablet for tablet 49b2a60664f1419e918bd9a4f5a67508 (DEFAULT_TABLE table=test-table [id=4fd3a5f7fdac452ebd2ea8fdad9e6943]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:04.738490 8239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49b2a60664f1419e918bd9a4f5a67508. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:04.738421 8378 tablet_service.cc:1467] Processing CreateTablet for tablet 49b2a60664f1419e918bd9a4f5a67508 (DEFAULT_TABLE table=test-table [id=4fd3a5f7fdac452ebd2ea8fdad9e6943]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:04.738866 8378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49b2a60664f1419e918bd9a4f5a67508. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:04.745568 8517 tablet_service.cc:1467] Processing CreateTablet for tablet 49b2a60664f1419e918bd9a4f5a67508 (DEFAULT_TABLE table=test-table [id=4fd3a5f7fdac452ebd2ea8fdad9e6943]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:04.746104 8517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49b2a60664f1419e918bd9a4f5a67508. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:04.751703 8607 tablet_bootstrap.cc:492] T 49b2a60664f1419e918bd9a4f5a67508 P e776fe34da26474480744b8040065764: Bootstrap starting.
I20260926 15:15:04.753702 8607 tablet_bootstrap.cc:654] T 49b2a60664f1419e918bd9a4f5a67508 P e776fe34da26474480744b8040065764: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:04.755546 8608 tablet_bootstrap.cc:492] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455: Bootstrap starting.
I20260926 15:15:04.756917 8608 tablet_bootstrap.cc:654] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:04.756968 8609 tablet_bootstrap.cc:492] T 49b2a60664f1419e918bd9a4f5a67508 P d976da13ccef419ea690cc6682cafebb: Bootstrap starting.
I20260926 15:15:04.758671 8609 tablet_bootstrap.cc:654] T 49b2a60664f1419e918bd9a4f5a67508 P d976da13ccef419ea690cc6682cafebb: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:04.770320 8608 tablet_bootstrap.cc:492] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455: No bootstrap required, opened a new log
I20260926 15:15:04.770466 8608 ts_tablet_manager.cc:1397] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20260926 15:15:04.771368 8608 raft_consensus.cc:348] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } }
I20260926 15:15:04.771602 8608 raft_consensus.cc:374] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:04.771683 8608 raft_consensus.cc:729] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 367b737530864214bdf43d409f5c7455, State: Initialized, Role: FOLLOWER
I20260926 15:15:04.771968 8609 tablet_bootstrap.cc:492] T 49b2a60664f1419e918bd9a4f5a67508 P d976da13ccef419ea690cc6682cafebb: No bootstrap required, opened a new log
I20260926 15:15:04.772104 8609 ts_tablet_manager.cc:1397] T 49b2a60664f1419e918bd9a4f5a67508 P d976da13ccef419ea690cc6682cafebb: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20260926 15:15:04.771960 8608 consensus_queue.cc:260] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } }
I20260926 15:15:04.772464 8608 ts_tablet_manager.cc:1428] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 15:15:04.772797 8607 tablet_bootstrap.cc:492] T 49b2a60664f1419e918bd9a4f5a67508 P e776fe34da26474480744b8040065764: No bootstrap required, opened a new log
I20260926 15:15:04.772775 8609 raft_consensus.cc:348] T 49b2a60664f1419e918bd9a4f5a67508 P d976da13ccef419ea690cc6682cafebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } }
I20260926 15:15:04.772962 8607 ts_tablet_manager.cc:1397] T 49b2a60664f1419e918bd9a4f5a67508 P e776fe34da26474480744b8040065764: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.004s
I20260926 15:15:04.773027 8609 raft_consensus.cc:374] T 49b2a60664f1419e918bd9a4f5a67508 P d976da13ccef419ea690cc6682cafebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:04.773129 8609 raft_consensus.cc:729] T 49b2a60664f1419e918bd9a4f5a67508 P d976da13ccef419ea690cc6682cafebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d976da13ccef419ea690cc6682cafebb, State: Initialized, Role: FOLLOWER
I20260926 15:15:04.773365 8609 consensus_queue.cc:260] T 49b2a60664f1419e918bd9a4f5a67508 P d976da13ccef419ea690cc6682cafebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } }
I20260926 15:15:04.773890 8609 ts_tablet_manager.cc:1428] T 49b2a60664f1419e918bd9a4f5a67508 P d976da13ccef419ea690cc6682cafebb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:15:04.774464 8632 tablet_replica.cc:330] T 978a090ca2fb440ca27c9c642c443710 P d976da13ccef419ea690cc6682cafebb: stopping tablet replica
I20260926 15:15:04.774837 8632 raft_consensus.cc:2229] T 978a090ca2fb440ca27c9c642c443710 P d976da13ccef419ea690cc6682cafebb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:04.776104 8632 raft_consensus.cc:2258] T 978a090ca2fb440ca27c9c642c443710 P d976da13ccef419ea690cc6682cafebb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 15:15:04.776528 8450 tablet.cc:2369] T 49b2a60664f1419e918bd9a4f5a67508 P d976da13ccef419ea690cc6682cafebb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:15:04.777503 8633 tablet_replica.cc:330] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764: stopping tablet replica
I20260926 15:15:04.777840 8633 raft_consensus.cc:2229] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:15:04.778291 8632 ts_tablet_manager.cc:1905] T 978a090ca2fb440ca27c9c642c443710 P d976da13ccef419ea690cc6682cafebb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:04.781589 8607 raft_consensus.cc:348] T 49b2a60664f1419e918bd9a4f5a67508 P e776fe34da26474480744b8040065764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } }
I20260926 15:15:04.781809 8607 raft_consensus.cc:374] T 49b2a60664f1419e918bd9a4f5a67508 P e776fe34da26474480744b8040065764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:04.781915 8607 raft_consensus.cc:729] T 49b2a60664f1419e918bd9a4f5a67508 P e776fe34da26474480744b8040065764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e776fe34da26474480744b8040065764, State: Initialized, Role: FOLLOWER
I20260926 15:15:04.782343 8633 raft_consensus.cc:2258] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:04.782765 8607 consensus_queue.cc:260] T 49b2a60664f1419e918bd9a4f5a67508 P e776fe34da26474480744b8040065764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } }
I20260926 15:15:04.783318 8607 ts_tablet_manager.cc:1428] T 49b2a60664f1419e918bd9a4f5a67508 P e776fe34da26474480744b8040065764: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20260926 15:15:04.786322 8632 ts_tablet_manager.cc:1918] T 978a090ca2fb440ca27c9c642c443710 P d976da13ccef419ea690cc6682cafebb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:15:04.786475 8632 log.cc:1196] T 978a090ca2fb440ca27c9c642c443710 P d976da13ccef419ea690cc6682cafebb: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/978a090ca2fb440ca27c9c642c443710
I20260926 15:15:04.786947 8632 ts_tablet_manager.cc:1939] T 978a090ca2fb440ca27c9c642c443710 P d976da13ccef419ea690cc6682cafebb: Deleting consensus metadata
I20260926 15:15:04.787778 8633 ts_tablet_manager.cc:1905] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:04.789285 8631 tablet_replica.cc:330] T 978a090ca2fb440ca27c9c642c443710 P 367b737530864214bdf43d409f5c7455: stopping tablet replica
I20260926 15:15:04.789358 8089 catalog_manager.cc:4841] TS d976da13ccef419ea690cc6682cafebb (127.6.209.194:36291): tablet 978a090ca2fb440ca27c9c642c443710 (table test-table [id=40f08a29a9874617b788487b926923ae]) successfully deleted
I20260926 15:15:04.789652 8631 raft_consensus.cc:2229] T 978a090ca2fb440ca27c9c642c443710 P 367b737530864214bdf43d409f5c7455 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:04.790014 8631 raft_consensus.cc:2258] T 978a090ca2fb440ca27c9c642c443710 P 367b737530864214bdf43d409f5c7455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:04.792126 8631 ts_tablet_manager.cc:1905] T 978a090ca2fb440ca27c9c642c443710 P 367b737530864214bdf43d409f5c7455: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:04.796216 8633 ts_tablet_manager.cc:1918] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:15:04.796348 8633 log.cc:1196] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-0/wal/wals/978a090ca2fb440ca27c9c642c443710
I20260926 15:15:04.796801 8633 ts_tablet_manager.cc:1939] T 978a090ca2fb440ca27c9c642c443710 P e776fe34da26474480744b8040065764: Deleting consensus metadata
I20260926 15:15:04.798246 8631 ts_tablet_manager.cc:1918] T 978a090ca2fb440ca27c9c642c443710 P 367b737530864214bdf43d409f5c7455: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:15:04.798234 8091 catalog_manager.cc:4841] TS e776fe34da26474480744b8040065764 (127.6.209.193:39569): tablet 978a090ca2fb440ca27c9c642c443710 (table test-table [id=40f08a29a9874617b788487b926923ae]) successfully deleted
I20260926 15:15:04.798380 8631 log.cc:1196] T 978a090ca2fb440ca27c9c642c443710 P 367b737530864214bdf43d409f5c7455: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/978a090ca2fb440ca27c9c642c443710
I20260926 15:15:04.798725 8631 ts_tablet_manager.cc:1939] T 978a090ca2fb440ca27c9c642c443710 P 367b737530864214bdf43d409f5c7455: Deleting consensus metadata
I20260926 15:15:04.799890 8089 catalog_manager.cc:4841] TS 367b737530864214bdf43d409f5c7455 (127.6.209.195:33525): tablet 978a090ca2fb440ca27c9c642c443710 (table test-table [id=40f08a29a9874617b788487b926923ae]) successfully deleted
W20260926 15:15:04.823643 8311 tablet.cc:2369] T 49b2a60664f1419e918bd9a4f5a67508 P e776fe34da26474480744b8040065764: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:15:04.924664 8614 raft_consensus.cc:482] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:15:04.924876 8614 raft_consensus.cc:504] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } }
I20260926 15:15:04.926626 8614 leader_election.cc:288] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e776fe34da26474480744b8040065764 (127.6.209.193:39569), d976da13ccef419ea690cc6682cafebb (127.6.209.194:36291)
I20260926 15:15:04.933056 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:04.926940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:39569 (local address 127.6.209.195:40647)
0926 15:15:04.927320 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:04.927352 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:04.927479 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:04.927981 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:04.927985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:04.928008 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:04.928186 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:04.928195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.929479 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.929484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:04.931314 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:04.931328 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.932382 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.932386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:04.932534 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:04.932555 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 15:15:04.932645 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 15:15:04.932752 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234}
I20260926 15:15:04.933086 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:04.927019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:40647 (local address 127.6.209.193:39569)
0926 15:15:04.927269 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:04.927280 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:04.927610 (+ 330us) server_negotiation.cc:407] Connection header received
0926 15:15:04.927677 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:04.927683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:04.927749 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:04.927818 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:04.928331 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.929372 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.931454 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.932257 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.932395 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:04.932457 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:04.932666 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 15:15:04.932770 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 15:15:04.933843 8262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b2a60664f1419e918bd9a4f5a67508" candidate_uuid: "367b737530864214bdf43d409f5c7455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e776fe34da26474480744b8040065764" is_pre_election: true
I20260926 15:15:04.934145 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:04.927433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:38989 (local address 127.6.209.194:36291)
0926 15:15:04.927601 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:04.927607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:04.927804 (+ 197us) server_negotiation.cc:407] Connection header received
0926 15:15:04.928085 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:04.928090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:04.928152 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:04.928272 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:04.928800 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.930075 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.931866 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.932856 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.933025 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:04.933087 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:04.933758 (+ 671us) server_negotiation.cc:299] Negotiation successful
0926 15:15:04.933878 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 15:15:04.934188 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:04.926969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:36291 (local address 127.6.209.195:38989)
0926 15:15:04.927709 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:04.927725 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:04.927926 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:04.928351 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:04.928355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:04.928372 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:04.928646 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:04.928654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.930183 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.930189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:04.931725 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:04.931738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:04.933448 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:04.933454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:04.933609 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:04.933632 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 15:15:04.933736 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 15:15:04.933849 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":635,"threads_started":1}
I20260926 15:15:04.934273 8262 raft_consensus.cc:2454] T 49b2a60664f1419e918bd9a4f5a67508 P e776fe34da26474480744b8040065764 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 367b737530864214bdf43d409f5c7455 in term 0.
I20260926 15:15:04.934808 8472 leader_election.cc:302] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 367b737530864214bdf43d409f5c7455, e776fe34da26474480744b8040065764; no voters:
I20260926 15:15:04.934816 8399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b2a60664f1419e918bd9a4f5a67508" candidate_uuid: "367b737530864214bdf43d409f5c7455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d976da13ccef419ea690cc6682cafebb" is_pre_election: true
I20260926 15:15:04.935014 8399 raft_consensus.cc:2454] T 49b2a60664f1419e918bd9a4f5a67508 P d976da13ccef419ea690cc6682cafebb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 367b737530864214bdf43d409f5c7455 in term 0.
I20260926 15:15:04.935140 8614 raft_consensus.cc:2781] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:15:04.935256 8614 raft_consensus.cc:482] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:15:04.935340 8614 raft_consensus.cc:3037] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:04.937188 8614 raft_consensus.cc:504] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } }
I20260926 15:15:04.937707 8614 leader_election.cc:288] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455 [CANDIDATE]: Term 1 election: Requested vote from peers e776fe34da26474480744b8040065764 (127.6.209.193:39569), d976da13ccef419ea690cc6682cafebb (127.6.209.194:36291)
I20260926 15:15:04.938069 8262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b2a60664f1419e918bd9a4f5a67508" candidate_uuid: "367b737530864214bdf43d409f5c7455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e776fe34da26474480744b8040065764"
I20260926 15:15:04.938184 8399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b2a60664f1419e918bd9a4f5a67508" candidate_uuid: "367b737530864214bdf43d409f5c7455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d976da13ccef419ea690cc6682cafebb"
I20260926 15:15:04.938263 8262 raft_consensus.cc:3037] T 49b2a60664f1419e918bd9a4f5a67508 P e776fe34da26474480744b8040065764 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:04.938355 8399 raft_consensus.cc:3037] T 49b2a60664f1419e918bd9a4f5a67508 P d976da13ccef419ea690cc6682cafebb [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:04.940901 8399 raft_consensus.cc:2454] T 49b2a60664f1419e918bd9a4f5a67508 P d976da13ccef419ea690cc6682cafebb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 367b737530864214bdf43d409f5c7455 in term 1.
I20260926 15:15:04.941284 8470 leader_election.cc:302] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 367b737530864214bdf43d409f5c7455, d976da13ccef419ea690cc6682cafebb; no voters:
I20260926 15:15:04.941569 8614 raft_consensus.cc:2781] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:04.941663 8262 raft_consensus.cc:2454] T 49b2a60664f1419e918bd9a4f5a67508 P e776fe34da26474480744b8040065764 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 367b737530864214bdf43d409f5c7455 in term 1.
I20260926 15:15:04.942094 8614 raft_consensus.cc:686] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455 [term 1 LEADER]: Becoming Leader. State: Replica: 367b737530864214bdf43d409f5c7455, State: Running, Role: LEADER
I20260926 15:15:04.942757 8614 consensus_queue.cc:237] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } }
I20260926 15:15:04.946280 8104 catalog_manager.cc:5495] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455 reported cstate change: term changed from 0 to 1, leader changed from <none> to 367b737530864214bdf43d409f5c7455 (127.6.209.195). New cstate: current_term: 1 leader_uuid: "367b737530864214bdf43d409f5c7455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } health_report { overall_health: HEALTHY } } }
W20260926 15:15:04.992847 8589 tablet.cc:2369] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 15:15:05 dist-test-slave-j6q7 krb5kdc[8059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435703, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:05.031490 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:05.010981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40837 (local address 127.0.0.1:47758)
0926 15:15:05.011577 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:05.011597 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:05.011760 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:05.012060 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:05.012064 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:05.013004 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:05.013310 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:05.013322 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:05.014411 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:05.014415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:05.015902 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:05.015916 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:05.016531 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:05.016537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:05.016703 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:05.022110 (+ 5407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:05.022133 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:05.024890 (+ 2757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:05.028839 (+ 3949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:05.028851 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:05.028858 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:05.029407 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:05.029785 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:05.029792 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:05.029797 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:05.029952 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:05.030563 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:05.030569 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:05.030741 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 15:15:05.030876 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 15:15:05.031107 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":328,"threads_started":1}
I20260926 15:15:05.031855 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:05.010967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47758 (local address 127.6.209.254:40837)
0926 15:15:05.011151 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:05.011159 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:05.011701 (+ 542us) server_negotiation.cc:407] Connection header received
0926 15:15:05.011864 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:05.011868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:05.011919 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:05.011994 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:05.013445 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:05.014299 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:05.016021 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:05.016366 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:05.016452 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:05.025057 (+ 8605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:05.025063 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:05.025067 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:05.025100 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:05.028687 (+ 3587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:05.029519 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:05.029524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:05.029527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:05.029669 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:05.030072 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:05.030076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:05.030079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:05.030430 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:05.030579 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:05.031013 (+ 434us) server_negotiation.cc:299] Negotiation successful
0926 15:15:05.031238 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":56}
I20260926 15:15:05.036839 8104 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:47758:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 15:15:05.037949 8104 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:15:05.044927 8240 tablet_service.cc:1467] Processing CreateTablet for tablet 2e9f970166b24c979aaef42204d62c12 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aea1e72311f84ae5877b81dcb9c33d05]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:15:05.045082 8518 tablet_service.cc:1467] Processing CreateTablet for tablet 2e9f970166b24c979aaef42204d62c12 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aea1e72311f84ae5877b81dcb9c33d05]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:15:05.045248 8379 tablet_service.cc:1467] Processing CreateTablet for tablet 2e9f970166b24c979aaef42204d62c12 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aea1e72311f84ae5877b81dcb9c33d05]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:15:05.045491 8240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e9f970166b24c979aaef42204d62c12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:05.045621 8518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e9f970166b24c979aaef42204d62c12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:05.045694 8379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e9f970166b24c979aaef42204d62c12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:05.052193 8607 tablet_bootstrap.cc:492] T 2e9f970166b24c979aaef42204d62c12 P e776fe34da26474480744b8040065764: Bootstrap starting.
I20260926 15:15:05.052217 8608 tablet_bootstrap.cc:492] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455: Bootstrap starting.
I20260926 15:15:05.052304 8609 tablet_bootstrap.cc:492] T 2e9f970166b24c979aaef42204d62c12 P d976da13ccef419ea690cc6682cafebb: Bootstrap starting.
I20260926 15:15:05.053856 8608 tablet_bootstrap.cc:654] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:05.053859 8609 tablet_bootstrap.cc:654] T 2e9f970166b24c979aaef42204d62c12 P d976da13ccef419ea690cc6682cafebb: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:05.053871 8607 tablet_bootstrap.cc:654] T 2e9f970166b24c979aaef42204d62c12 P e776fe34da26474480744b8040065764: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:05.063254 8609 tablet_bootstrap.cc:492] T 2e9f970166b24c979aaef42204d62c12 P d976da13ccef419ea690cc6682cafebb: No bootstrap required, opened a new log
I20260926 15:15:05.063452 8609 ts_tablet_manager.cc:1397] T 2e9f970166b24c979aaef42204d62c12 P d976da13ccef419ea690cc6682cafebb: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.006s
I20260926 15:15:05.064252 8609 raft_consensus.cc:348] T 2e9f970166b24c979aaef42204d62c12 P d976da13ccef419ea690cc6682cafebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } }
I20260926 15:15:05.064384 8608 tablet_bootstrap.cc:492] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455: No bootstrap required, opened a new log
I20260926 15:15:05.064467 8609 raft_consensus.cc:374] T 2e9f970166b24c979aaef42204d62c12 P d976da13ccef419ea690cc6682cafebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:05.064533 8609 raft_consensus.cc:729] T 2e9f970166b24c979aaef42204d62c12 P d976da13ccef419ea690cc6682cafebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d976da13ccef419ea690cc6682cafebb, State: Initialized, Role: FOLLOWER
I20260926 15:15:05.064782 8609 consensus_queue.cc:260] T 2e9f970166b24c979aaef42204d62c12 P d976da13ccef419ea690cc6682cafebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } }
I20260926 15:15:05.065161 8607 tablet_bootstrap.cc:492] T 2e9f970166b24c979aaef42204d62c12 P e776fe34da26474480744b8040065764: No bootstrap required, opened a new log
I20260926 15:15:05.065289 8607 ts_tablet_manager.cc:1397] T 2e9f970166b24c979aaef42204d62c12 P e776fe34da26474480744b8040065764: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20260926 15:15:05.065344 8609 ts_tablet_manager.cc:1428] T 2e9f970166b24c979aaef42204d62c12 P d976da13ccef419ea690cc6682cafebb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:15:05.065240 8613 tablet_replica.cc:439] T 2e9f970166b24c979aaef42204d62c12 P d976da13ccef419ea690cc6682cafebb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } } }
I20260926 15:15:05.065573 8613 tablet_replica.cc:442] T 2e9f970166b24c979aaef42204d62c12 P d976da13ccef419ea690cc6682cafebb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:15:05.065789 8607 raft_consensus.cc:348] T 2e9f970166b24c979aaef42204d62c12 P e776fe34da26474480744b8040065764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } }
I20260926 15:15:05.065928 8607 raft_consensus.cc:374] T 2e9f970166b24c979aaef42204d62c12 P e776fe34da26474480744b8040065764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:05.065973 8607 raft_consensus.cc:729] T 2e9f970166b24c979aaef42204d62c12 P e776fe34da26474480744b8040065764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e776fe34da26474480744b8040065764, State: Initialized, Role: FOLLOWER
I20260926 15:15:05.066828 8607 consensus_queue.cc:260] T 2e9f970166b24c979aaef42204d62c12 P e776fe34da26474480744b8040065764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } }
I20260926 15:15:05.067255 8626 tablet_replica.cc:439] T 2e9f970166b24c979aaef42204d62c12 P e776fe34da26474480744b8040065764: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } } }
I20260926 15:15:05.067521 8626 tablet_replica.cc:442] T 2e9f970166b24c979aaef42204d62c12 P e776fe34da26474480744b8040065764: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:15:05.068375 8607 ts_tablet_manager.cc:1428] T 2e9f970166b24c979aaef42204d62c12 P e776fe34da26474480744b8040065764: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 15:15:05.068625 8608 ts_tablet_manager.cc:1397] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.009s
I20260926 15:15:05.069232 8608 raft_consensus.cc:348] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } }
I20260926 15:15:05.069427 8608 raft_consensus.cc:374] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:05.069482 8608 raft_consensus.cc:729] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 367b737530864214bdf43d409f5c7455, State: Initialized, Role: FOLLOWER
I20260926 15:15:05.069653 8608 consensus_queue.cc:260] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } }
I20260926 15:15:05.069969 8614 tablet_replica.cc:439] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } } }
I20260926 15:15:05.070199 8614 tablet_replica.cc:442] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:15:05.070515 8608 ts_tablet_manager.cc:1428] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260926 15:15:05.096025 8614 raft_consensus.cc:482] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:15:05.096186 8614 raft_consensus.cc:504] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } }
I20260926 15:15:05.096721 8614 leader_election.cc:288] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e776fe34da26474480744b8040065764 (127.6.209.193:39569), d976da13ccef419ea690cc6682cafebb (127.6.209.194:36291)
I20260926 15:15:05.097144 8399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9f970166b24c979aaef42204d62c12" candidate_uuid: "367b737530864214bdf43d409f5c7455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d976da13ccef419ea690cc6682cafebb" is_pre_election: true
I20260926 15:15:05.097194 8262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9f970166b24c979aaef42204d62c12" candidate_uuid: "367b737530864214bdf43d409f5c7455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e776fe34da26474480744b8040065764" is_pre_election: true
I20260926 15:15:05.097395 8399 raft_consensus.cc:2454] T 2e9f970166b24c979aaef42204d62c12 P d976da13ccef419ea690cc6682cafebb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 367b737530864214bdf43d409f5c7455 in term 0.
I20260926 15:15:05.097414 8262 raft_consensus.cc:2454] T 2e9f970166b24c979aaef42204d62c12 P e776fe34da26474480744b8040065764 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 367b737530864214bdf43d409f5c7455 in term 0.
I20260926 15:15:05.097741 8472 leader_election.cc:302] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 367b737530864214bdf43d409f5c7455, e776fe34da26474480744b8040065764; no voters:
I20260926 15:15:05.097993 8614 raft_consensus.cc:2781] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:15:05.098105 8614 raft_consensus.cc:482] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:15:05.098215 8614 raft_consensus.cc:3037] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:05.100423 8614 raft_consensus.cc:504] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } }
I20260926 15:15:05.101030 8614 leader_election.cc:288] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455 [CANDIDATE]: Term 1 election: Requested vote from peers e776fe34da26474480744b8040065764 (127.6.209.193:39569), d976da13ccef419ea690cc6682cafebb (127.6.209.194:36291)
I20260926 15:15:05.101342 8262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9f970166b24c979aaef42204d62c12" candidate_uuid: "367b737530864214bdf43d409f5c7455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e776fe34da26474480744b8040065764"
I20260926 15:15:05.101559 8262 raft_consensus.cc:3037] T 2e9f970166b24c979aaef42204d62c12 P e776fe34da26474480744b8040065764 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:05.101599 8399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9f970166b24c979aaef42204d62c12" candidate_uuid: "367b737530864214bdf43d409f5c7455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d976da13ccef419ea690cc6682cafebb"
I20260926 15:15:05.101840 8399 raft_consensus.cc:3037] T 2e9f970166b24c979aaef42204d62c12 P d976da13ccef419ea690cc6682cafebb [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:05.104027 8262 raft_consensus.cc:2454] T 2e9f970166b24c979aaef42204d62c12 P e776fe34da26474480744b8040065764 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 367b737530864214bdf43d409f5c7455 in term 1.
I20260926 15:15:05.104416 8472 leader_election.cc:302] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 367b737530864214bdf43d409f5c7455, e776fe34da26474480744b8040065764; no voters:
I20260926 15:15:05.104676 8614 raft_consensus.cc:2781] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:05.104858 8614 raft_consensus.cc:686] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455 [term 1 LEADER]: Becoming Leader. State: Replica: 367b737530864214bdf43d409f5c7455, State: Running, Role: LEADER
I20260926 15:15:05.104902 8399 raft_consensus.cc:2454] T 2e9f970166b24c979aaef42204d62c12 P d976da13ccef419ea690cc6682cafebb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 367b737530864214bdf43d409f5c7455 in term 1.
I20260926 15:15:05.105086 8614 consensus_queue.cc:237] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } }
I20260926 15:15:05.106679 8638 tablet_replica.cc:439] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455: TxnStatusTablet state changed. Reason: New leader 367b737530864214bdf43d409f5c7455. Latest consensus state: current_term: 1 leader_uuid: "367b737530864214bdf43d409f5c7455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } } }
I20260926 15:15:05.106979 8638 tablet_replica.cc:442] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:15:05.107640 8650 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 15:15:05.107860 8650 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 15:15:05.108741 8101 catalog_manager.cc:5495] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455 reported cstate change: term changed from 0 to 1, leader changed from <none> to 367b737530864214bdf43d409f5c7455 (127.6.209.195). New cstate: current_term: 1 leader_uuid: "367b737530864214bdf43d409f5c7455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } health_report { overall_health: HEALTHY } } }
I20260926 15:15:05.128347 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:05.121589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:33525 (local address 127.0.0.1:56586)
0926 15:15:05.121823 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:05.121844 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:05.121984 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:05.122533 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:05.122538 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:05.122551 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:15:05.122736 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:05.122746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:05.124190 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:05.124196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:05.125741 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:05.125754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:05.126668 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:05.126673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:05.126860 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:05.126929 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:15:05.127792 (+ 863us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:15:05.127811 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 15:15:05.127891 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 15:15:05.128022 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 15:15:05.128409 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:05.121943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56586 (local address 127.6.209.195:33525)
0926 15:15:05.122111 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:05.122117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:05.122141 (+ 24us) server_negotiation.cc:407] Connection header received
0926 15:15:05.122231 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:05.122240 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:05.122347 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:05.122502 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:15:05.122889 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:05.124055 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:05.125890 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:05.126506 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:05.126636 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:05.127095 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:15:05.127680 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:15:05.127808 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:05.127916 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 15:15:05.128058 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 15:15:05.132885 8262 raft_consensus.cc:1261] T 2e9f970166b24c979aaef42204d62c12 P e776fe34da26474480744b8040065764 [term 1 FOLLOWER]: Refusing update from remote peer 367b737530864214bdf43d409f5c7455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:05.133246 8399 raft_consensus.cc:1261] T 2e9f970166b24c979aaef42204d62c12 P d976da13ccef419ea690cc6682cafebb [term 1 FOLLOWER]: Refusing update from remote peer 367b737530864214bdf43d409f5c7455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:05.133730 8614 consensus_queue.cc:1035] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:05.134326 8614 consensus_queue.cc:1035] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:05.146123 8626 tablet_replica.cc:439] T 2e9f970166b24c979aaef42204d62c12 P e776fe34da26474480744b8040065764: TxnStatusTablet state changed. Reason: New leader 367b737530864214bdf43d409f5c7455. Latest consensus state: current_term: 1 leader_uuid: "367b737530864214bdf43d409f5c7455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } } }
I20260926 15:15:05.146836 8626 tablet_replica.cc:442] T 2e9f970166b24c979aaef42204d62c12 P e776fe34da26474480744b8040065764: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:15:05.152100 8623 mvcc.cc:205] Tried to move back new op lower bound from 7333624648216866816 to 7333624648112701440. Current Snapshot: MvccSnapshot[applied={T|T < 7333624648216866816 or (T in {7333624648216866816})}]
I20260926 15:15:05.159164 8621 mvcc.cc:205] Tried to move back new op lower bound from 7333624648216866816 to 7333624648112701440. Current Snapshot: MvccSnapshot[applied={T|T < 7333624648216866816}]
I20260926 15:15:05.161250 8626 tablet_replica.cc:439] T 2e9f970166b24c979aaef42204d62c12 P e776fe34da26474480744b8040065764: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "367b737530864214bdf43d409f5c7455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } } }
I20260926 15:15:05.161485 8626 tablet_replica.cc:442] T 2e9f970166b24c979aaef42204d62c12 P e776fe34da26474480744b8040065764: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:15:05.150244 8638 tablet_replica.cc:439] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "367b737530864214bdf43d409f5c7455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } } }
I20260926 15:15:05.164542 8638 tablet_replica.cc:442] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:15:05.171496 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:05.158119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.6.209.195:33525)
0926 15:15:05.164775 (+ 6656us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:05.164781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:05.164798 (+ 17us) server_negotiation.cc:407] Connection header received
0926 15:15:05.165330 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:05.165336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:05.165418 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:05.165946 (+ 528us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:15:05.166068 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:05.167058 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:05.168952 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:05.169632 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:05.170226 (+ 594us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:05.170356 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:15:05.170467 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:15:05.171077 (+ 610us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:05.171129 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 15:15:05.171198 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6542}
I20260926 15:15:05.173558 8613 tablet_replica.cc:439] T 2e9f970166b24c979aaef42204d62c12 P d976da13ccef419ea690cc6682cafebb: TxnStatusTablet state changed. Reason: New leader 367b737530864214bdf43d409f5c7455. Latest consensus state: current_term: 1 leader_uuid: "367b737530864214bdf43d409f5c7455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } } }
I20260926 15:15:05.173837 8613 tablet_replica.cc:442] T 2e9f970166b24c979aaef42204d62c12 P d976da13ccef419ea690cc6682cafebb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:15:05.176540 8638 tablet_replica.cc:439] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "367b737530864214bdf43d409f5c7455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } } }
I20260926 15:15:05.176749 8638 tablet_replica.cc:442] T 2e9f970166b24c979aaef42204d62c12 P 367b737530864214bdf43d409f5c7455: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:15:05.177973 8613 tablet_replica.cc:439] T 2e9f970166b24c979aaef42204d62c12 P d976da13ccef419ea690cc6682cafebb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "367b737530864214bdf43d409f5c7455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 } } peers { permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 } } peers { permanent_uuid: "367b737530864214bdf43d409f5c7455" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 33525 } } }
I20260926 15:15:05.178218 8613 tablet_replica.cc:442] T 2e9f970166b24c979aaef42204d62c12 P d976da13ccef419ea690cc6682cafebb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:15:05.192651 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:05.186541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:44847 (local address 127.6.209.195:33525)
0926 15:15:05.186953 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:05.186959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:05.187298 (+ 339us) server_negotiation.cc:407] Connection header received
0926 15:15:05.187436 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:05.187441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:05.187520 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:05.187606 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:15:05.188119 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:05.189112 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:05.190897 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:05.191505 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:05.191601 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:05.192031 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:15:05.192140 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:15:05.192209 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:05.192349 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 15:15:05.192414 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":306}
I20260926 15:15:05.192842 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:05.186656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:33525 (local address 127.6.209.195:44847)
0926 15:15:05.187205 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:05.187222 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:05.187334 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:05.187631 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:05.187636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:05.187646 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:15:05.187964 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:05.187974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:05.189239 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:05.189244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:05.190729 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:05.190742 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:05.191672 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:05.191678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:05.191832 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:05.191887 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:15:05.192232 (+ 345us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:15:05.192250 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 15:15:05.192325 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 15:15:05.192449 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":3,"thread_start_us":306,"threads_started":1}
I20260926 15:15:05.200809 8262 raft_consensus.cc:1261] T 49b2a60664f1419e918bd9a4f5a67508 P e776fe34da26474480744b8040065764 [term 1 FOLLOWER]: Refusing update from remote peer 367b737530864214bdf43d409f5c7455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:05.201099 8399 raft_consensus.cc:1261] T 49b2a60664f1419e918bd9a4f5a67508 P d976da13ccef419ea690cc6682cafebb [term 1 FOLLOWER]: Refusing update from remote peer 367b737530864214bdf43d409f5c7455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:05.201440 8614 consensus_queue.cc:1035] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e776fe34da26474480744b8040065764" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 39569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:05.201711 8638 consensus_queue.cc:1035] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d976da13ccef419ea690cc6682cafebb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:05.271009 8104 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:05.271206 8104 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:05.273671 8104 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 911f623fb5f545b4972ec63846443d40: Sending DeleteTablet for 3 replicas of tablet 49b2a60664f1419e918bd9a4f5a67508
I20260926 15:15:05.274312 8240 tablet_service.cc:1514] Processing DeleteTablet for tablet 49b2a60664f1419e918bd9a4f5a67508 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:05 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:41510
I20260926 15:15:05.274564 8633 tablet_replica.cc:330] T 49b2a60664f1419e918bd9a4f5a67508 P e776fe34da26474480744b8040065764: stopping tablet replica
I20260926 15:15:05.274775 8633 raft_consensus.cc:2229] T 49b2a60664f1419e918bd9a4f5a67508 P e776fe34da26474480744b8040065764 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:05.274727 8379 tablet_service.cc:1514] Processing DeleteTablet for tablet 49b2a60664f1419e918bd9a4f5a67508 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:05 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:45242
I20260926 15:15:05.274950 8633 raft_consensus.cc:2258] T 49b2a60664f1419e918bd9a4f5a67508 P e776fe34da26474480744b8040065764 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:05.274982 8632 tablet_replica.cc:330] T 49b2a60664f1419e918bd9a4f5a67508 P d976da13ccef419ea690cc6682cafebb: stopping tablet replica
I20260926 15:15:05.275142 8632 raft_consensus.cc:2229] T 49b2a60664f1419e918bd9a4f5a67508 P d976da13ccef419ea690cc6682cafebb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:05.275281 8632 raft_consensus.cc:2258] T 49b2a60664f1419e918bd9a4f5a67508 P d976da13ccef419ea690cc6682cafebb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:05.275786 8633 ts_tablet_manager.cc:1905] T 49b2a60664f1419e918bd9a4f5a67508 P e776fe34da26474480744b8040065764: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:05.276150 8632 ts_tablet_manager.cc:1905] T 49b2a60664f1419e918bd9a4f5a67508 P d976da13ccef419ea690cc6682cafebb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:05.282974 8632 ts_tablet_manager.cc:1918] T 49b2a60664f1419e918bd9a4f5a67508 P d976da13ccef419ea690cc6682cafebb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:15:05.283119 8632 log.cc:1196] T 49b2a60664f1419e918bd9a4f5a67508 P d976da13ccef419ea690cc6682cafebb: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/49b2a60664f1419e918bd9a4f5a67508
I20260926 15:15:05.283514 8632 ts_tablet_manager.cc:1939] T 49b2a60664f1419e918bd9a4f5a67508 P d976da13ccef419ea690cc6682cafebb: Deleting consensus metadata
I20260926 15:15:05.284037 8633 ts_tablet_manager.cc:1918] T 49b2a60664f1419e918bd9a4f5a67508 P e776fe34da26474480744b8040065764: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:15:05.284262 8633 log.cc:1196] T 49b2a60664f1419e918bd9a4f5a67508 P e776fe34da26474480744b8040065764: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-0/wal/wals/49b2a60664f1419e918bd9a4f5a67508
I20260926 15:15:05.284698 8633 ts_tablet_manager.cc:1939] T 49b2a60664f1419e918bd9a4f5a67508 P e776fe34da26474480744b8040065764: Deleting consensus metadata
I20260926 15:15:05.284783 8089 catalog_manager.cc:4841] TS d976da13ccef419ea690cc6682cafebb (127.6.209.194:36291): tablet 49b2a60664f1419e918bd9a4f5a67508 (table test-table [id=4fd3a5f7fdac452ebd2ea8fdad9e6943]) successfully deleted
I20260926 15:15:05.284864 8518 tablet_service.cc:1514] Processing DeleteTablet for tablet 49b2a60664f1419e918bd9a4f5a67508 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:05 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:56574
I20260926 15:15:05.285123 8631 tablet_replica.cc:330] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455: stopping tablet replica
I20260926 15:15:05.285311 8631 raft_consensus.cc:2229] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:15:05.285701 8631 raft_consensus.cc:2258] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:05.285924 8091 catalog_manager.cc:4841] TS e776fe34da26474480744b8040065764 (127.6.209.193:39569): tablet 49b2a60664f1419e918bd9a4f5a67508 (table test-table [id=4fd3a5f7fdac452ebd2ea8fdad9e6943]) successfully deleted
I20260926 15:15:05.286700 8631 ts_tablet_manager.cc:1905] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:05.292846 8631 ts_tablet_manager.cc:1918] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:15:05.292984 8631 log.cc:1196] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/49b2a60664f1419e918bd9a4f5a67508
I20260926 15:15:05.293412 8631 ts_tablet_manager.cc:1939] T 49b2a60664f1419e918bd9a4f5a67508 P 367b737530864214bdf43d409f5c7455: Deleting consensus metadata
I20260926 15:15:05.294346 8089 catalog_manager.cc:4841] TS 367b737530864214bdf43d409f5c7455 (127.6.209.195:33525): tablet 49b2a60664f1419e918bd9a4f5a67508 (table test-table [id=4fd3a5f7fdac452ebd2ea8fdad9e6943]) successfully deleted
Sep 26 15:15:05 dist-test-slave-j6q7 krb5kdc[8059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435704, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.209.193@KRBTEST.COM
I20260926 15:15:05.301230 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:05.286930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41526 (local address 127.6.209.193:39569)
0926 15:15:05.287077 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:05.287083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:05.287633 (+ 550us) server_negotiation.cc:407] Connection header received
0926 15:15:05.287805 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:05.287809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:05.287881 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:05.287977 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:05.289286 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:05.290007 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:05.291177 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:05.291534 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:05.291647 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:05.296167 (+ 4520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:05.296176 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:05.296206 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:05.296235 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:05.298680 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:05.299406 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:05.299412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:05.299414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:05.299512 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:05.299854 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:05.299858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:05.299861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:05.300183 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:05.300371 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:05.300661 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 15:15:05.300879 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 15:15:05.301934 8305 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41526
I20260926 15:15:05.317077 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:05.304671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47774 (local address 127.6.209.254:40837)
0926 15:15:05.304903 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:05.304908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:05.305180 (+ 272us) server_negotiation.cc:407] Connection header received
0926 15:15:05.305334 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:05.305339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:05.305405 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:05.305493 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:05.307077 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:05.308114 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:05.309254 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:05.309842 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:05.309952 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:05.311720 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:05.311726 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:05.311731 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:05.311764 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:05.314862 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:05.315397 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:05.315401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:05.315404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:05.315522 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:05.315795 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:05.315799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:05.315803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:05.316096 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:05.316327 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:05.316492 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 15:15:05.316683 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 15:15:05.317600 8104 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47774
I20260926 15:15:05.320894 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 8175
I20260926 15:15:05.336447 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:05.321724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41534 (local address 127.6.209.193:39569)
0926 15:15:05.321918 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:05.321926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:05.326961 (+ 5035us) server_negotiation.cc:407] Connection header received
0926 15:15:05.327090 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:05.327095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:05.327158 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:05.327258 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:05.328324 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:05.329036 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:05.330003 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:05.330347 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:05.330461 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:05.332162 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:05.332168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:05.332171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:05.332207 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:05.334425 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:05.334926 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:05.334931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:05.334934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:05.335056 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:05.335354 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:05.335359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:05.335362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:05.335559 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:05.335754 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:05.335986 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 15:15:05.336175 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 15:15:05.337064 8305 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41534
W20260926 15:15:05.337531 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 8175: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:15:05.337680 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 8175
I20260926 15:15:05.357851 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 8314
Sep 26 15:15:05 dist-test-slave-j6q7 krb5kdc[8059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435704, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.209.194@KRBTEST.COM
I20260926 15:15:05.378691 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:05.358468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45244 (local address 127.6.209.194:36291)
0926 15:15:05.358620 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:05.358625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:05.358642 (+ 17us) server_negotiation.cc:407] Connection header received
0926 15:15:05.358726 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:05.358732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:05.358801 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:05.358927 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:05.359873 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:05.360885 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:05.362039 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:05.362611 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:05.362709 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:05.372698 (+ 9989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:05.372715 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:05.372730 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:05.372769 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:05.376206 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:05.376786 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:05.376794 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:05.376798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:05.377001 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:05.377300 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:05.377305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:05.377307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:05.377702 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:05.377977 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:05.378111 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 15:15:05.378331 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 15:15:05.379280 8444 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45244
W20260926 15:15:05.379846 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 8314: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:15:05.380004 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 8314
I20260926 15:15:05.395910 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 8453
Sep 26 15:15:05 dist-test-slave-j6q7 krb5kdc[8059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435704, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.209.195@KRBTEST.COM
I20260926 15:15:05.410002 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:05.396539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56598 (local address 127.6.209.195:33525)
0926 15:15:05.396701 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:05.396708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:05.396745 (+ 37us) server_negotiation.cc:407] Connection header received
0926 15:15:05.396929 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:05.396933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:05.396993 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:05.397089 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:05.398116 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:05.398803 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:05.399872 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:05.400319 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:05.400440 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:05.404394 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:05.404403 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:05.404413 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:05.404438 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:05.407565 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:05.408107 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:05.408115 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:05.408119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:05.408271 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:05.408610 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:05.408626 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:05.408629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:05.409046 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:05.409349 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:05.409481 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 15:15:05.409683 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 15:15:05.410640 8583 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56598
W20260926 15:15:05.411240 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 8453: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:15:05.411379 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 8453
I20260926 15:15:05.429661 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 8072
I20260926 15:15:05.430509 8144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:05.552160 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 8072
2026-09-26T15:15:05Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4020 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:15:05 dist-test-slave-j6q7 krb5kdc[8672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:15:05 dist-test-slave-j6q7 krb5kdc[8672](info): set up 2 sockets
Sep 26 15:15:05 dist-test-slave-j6q7 krb5kdc[8672](info): commencing operation
krb5kdc: starting...
W20260926 15:15:06.564742 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:15:06 dist-test-slave-j6q7 krb5kdc[8672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:15:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:15:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:15:06.719832 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:37989
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:37989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:07.000190 8685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:07.000428 8685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:07.000504 8685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:07.006824 8685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:15:07.006899 8685 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 15:15:07.006965 8685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:07.007041 8685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:07.007099 8685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:15:07.007191 8685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:15:07.015240 8685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:37989
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:37989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:07.016263 8685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:07.017498 8685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:07.027709 8691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:07.027736 8690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:07.027770 8693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:07.028327 8685 server_base.cc:1034] running on GCE node
I20260926 15:15:07.029011 8685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:07.030108 8685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:07.031319 8685 hybrid_clock.cc:648] HybridClock initialized: now 1790435707031272 us; error 50 us; skew 500 ppm
I20260926 15:15:07.031590 8685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:07 dist-test-slave-j6q7 krb5kdc[8672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435707, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:07.034387 8685 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:15:07.035561 8685 webserver.cc:460] Webserver started at http://127.6.209.254:36655/ using document root <none> and password file <none>
I20260926 15:15:07.036127 8685 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:07.036239 8685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:07.036513 8685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:07.038980 8685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "2a9b105f7a9749c985c9d52d0f08e7b1"
format_stamp: "Formatted at 2026-09-26 15:15:07 on dist-test-slave-j6q7"
server_key: "e8991c70aa7939fce97b69b0d475cb7f"
server_key_iv: "4222ca98b02813df61fa7913851b2fc0"
server_key_version: "encryptionkey@0"
I20260926 15:15:07.039645 8685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "2a9b105f7a9749c985c9d52d0f08e7b1"
format_stamp: "Formatted at 2026-09-26 15:15:07 on dist-test-slave-j6q7"
server_key: "e8991c70aa7939fce97b69b0d475cb7f"
server_key_iv: "4222ca98b02813df61fa7913851b2fc0"
server_key_version: "encryptionkey@0"
I20260926 15:15:07.044081 8685 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 15:15:07.046929 8700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:07.048074 8685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:15:07.048226 8685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "2a9b105f7a9749c985c9d52d0f08e7b1"
format_stamp: "Formatted at 2026-09-26 15:15:07 on dist-test-slave-j6q7"
server_key: "e8991c70aa7939fce97b69b0d475cb7f"
server_key_iv: "4222ca98b02813df61fa7913851b2fc0"
server_key_version: "encryptionkey@0"
I20260926 15:15:07.048410 8685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:07.069516 8685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:07.072657 8685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:07.072919 8685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:07.091450 8685 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:37989
I20260926 15:15:07.091471 8761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:37989 every 8 connection(s)
I20260926 15:15:07.092519 8685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:15:07.093567 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 8685
I20260926 15:15:07.093780 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:15:07.094569 6983 external_mini_cluster.cc:1428] Setting key c2b3365a805313d6c351439afe5fe155
I20260926 15:15:07.096673 8762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 15:15:07 dist-test-slave-j6q7 krb5kdc[8672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:07.108675 8762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1: Bootstrap starting.
I20260926 15:15:07.112308 8762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:07.113504 8762 log.cc:824] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:07.115962 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:07.096273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33458 (local address 127.6.209.254:37989)
0926 15:15:07.096954 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:07.096963 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:07.096998 (+ 35us) server_negotiation.cc:407] Connection header received
0926 15:15:07.098366 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:07.098400 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:07.098719 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:07.098994 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:07.100584 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:07.101930 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:07.103636 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:07.104223 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:07.104349 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:07.108524 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:07.108543 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:07.108557 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:07.108596 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:07.111945 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:07.112619 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:07.112630 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:07.112641 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:07.112834 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:07.113183 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:07.113192 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:07.113197 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:07.113837 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:07.114149 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:07.114518 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 15:15:07.114841 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":329,"threads_started":1}
I20260926 15:15:07.116428 8762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1: No bootstrap required, opened a new log
I20260926 15:15:07.123453 8762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9b105f7a9749c985c9d52d0f08e7b1" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37989 } }
I20260926 15:15:07.123728 8762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:07.123795 8762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a9b105f7a9749c985c9d52d0f08e7b1, State: Initialized, Role: FOLLOWER
I20260926 15:15:07.124280 8762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9b105f7a9749c985c9d52d0f08e7b1" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37989 } }
I20260926 15:15:07.124471 8762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:15:07.124557 8762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:15:07.124665 8762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:07.126444 8762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9b105f7a9749c985c9d52d0f08e7b1" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37989 } }
I20260926 15:15:07.126796 8762 leader_election.cc:302] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a9b105f7a9749c985c9d52d0f08e7b1; no voters:
I20260926 15:15:07.127381 8762 leader_election.cc:288] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:15:07.127473 8767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:07.128129 8767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1 [term 1 LEADER]: Becoming Leader. State: Replica: 2a9b105f7a9749c985c9d52d0f08e7b1, State: Running, Role: LEADER
I20260926 15:15:07.128598 8767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9b105f7a9749c985c9d52d0f08e7b1" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37989 } }
I20260926 15:15:07.129081 8762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:15:07.130941 8769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a9b105f7a9749c985c9d52d0f08e7b1. Latest consensus state: current_term: 1 leader_uuid: "2a9b105f7a9749c985c9d52d0f08e7b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9b105f7a9749c985c9d52d0f08e7b1" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37989 } } }
I20260926 15:15:07.130919 8768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a9b105f7a9749c985c9d52d0f08e7b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9b105f7a9749c985c9d52d0f08e7b1" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37989 } } }
I20260926 15:15:07.131381 8768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:07.131412 8769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:07.137152 8772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:15:07.141600 8772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:15:07.150133 8772 catalog_manager.cc:1347] Generated new cluster ID: 188fbcb47e3343d0862cba72e29743a2
I20260926 15:15:07.150230 8772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:15:07.161147 8772 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:15:07.162281 8772 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:15:07.167785 8772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1: Generated new TSK 0
I20260926 15:15:07.168385 8772 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
I20260926 15:15:07.216470 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:37989
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:07.516116 8788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:07.516422 8788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:07.516604 8788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:07.523301 8788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:07.523454 8788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:07.523638 8788 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 15:15:07.523736 8788 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:15:07.523861 8788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:15:07.532699 8788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.209.254:37989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:07.533900 8788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:07.535127 8788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:07.547305 8796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:07.547312 8793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:07.547276 8794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:07.547962 8788 server_base.cc:1034] running on GCE node
I20260926 15:15:07.548451 8788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:07.549353 8788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:07.550570 8788 hybrid_clock.cc:648] HybridClock initialized: now 1790435707550501 us; error 54 us; skew 500 ppm
I20260926 15:15:07.550841 8788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:07 dist-test-slave-j6q7 krb5kdc[8672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435707, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:07.553658 8788 init.cc:377] Logged in from keytab as kudu/127.6.209.193@KRBTEST.COM (short username kudu)
I20260926 15:15:07.555032 8788 webserver.cc:460] Webserver started at http://127.6.209.193:44347/ using document root <none> and password file <none>
I20260926 15:15:07.555647 8788 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:07.555781 8788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:07.556059 8788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:07.558713 8788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "9caa979a9710444182e0a168d483940e"
format_stamp: "Formatted at 2026-09-26 15:15:07 on dist-test-slave-j6q7"
server_key: "8543bbc8dc76d7b16017506e435597ac"
server_key_iv: "96ff9165ce224f33a0cad209849254e7"
server_key_version: "encryptionkey@0"
I20260926 15:15:07.559464 8788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "9caa979a9710444182e0a168d483940e"
format_stamp: "Formatted at 2026-09-26 15:15:07 on dist-test-slave-j6q7"
server_key: "8543bbc8dc76d7b16017506e435597ac"
server_key_iv: "96ff9165ce224f33a0cad209849254e7"
server_key_version: "encryptionkey@0"
I20260926 15:15:07.564337 8788 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 15:15:07.567444 8803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:07.568501 8788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 15:15:07.568668 8788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "9caa979a9710444182e0a168d483940e"
format_stamp: "Formatted at 2026-09-26 15:15:07 on dist-test-slave-j6q7"
server_key: "8543bbc8dc76d7b16017506e435597ac"
server_key_iv: "96ff9165ce224f33a0cad209849254e7"
server_key_version: "encryptionkey@0"
I20260926 15:15:07.568892 8788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:07.588450 8788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:07.591912 8788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:07.592188 8788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:07.594758 8788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:07.594897 8788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:07.595082 8788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:07.595206 8788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 15:15:07 dist-test-slave-j6q7 krb5kdc[8672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435707, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:07.616034 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:07.595034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:33277 (local address 127.6.209.254:37989)
0926 15:15:07.595243 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:07.595250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:07.596555 (+ 1305us) server_negotiation.cc:407] Connection header received
0926 15:15:07.598407 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:07.598417 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:07.598524 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:07.598634 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:07.600115 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:07.600997 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:07.603095 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:07.603959 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:07.604124 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:07.608664 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:07.608669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:07.608674 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:07.608713 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:07.612713 (+ 4000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:07.613580 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:07.613588 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:07.613592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:07.613740 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:07.614147 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:07.614152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:07.614155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:07.614415 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:07.614551 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:07.615218 (+ 667us) server_negotiation.cc:299] Negotiation successful
0926 15:15:07.615505 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 15:15:07.616408 8812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:07.595560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:37989 (local address 127.6.209.193:33277)
0926 15:15:07.596372 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:07.596427 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:07.598160 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:07.598827 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:07.598833 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:07.599243 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:07.599959 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:07.599976 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:07.601193 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:07.601196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:07.602935 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:07.602952 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:07.604320 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:07.604330 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:07.604553 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:07.605229 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:07.605270 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:07.608389 (+ 3119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:07.612899 (+ 4510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:07.612912 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:07.612925 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:07.613434 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:07.613854 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:07.613861 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:07.613864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:07.614034 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:07.614538 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:07.614544 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:07.614832 (+ 288us) client_negotiation.cc:769] Sending connection context
0926 15:15:07.615102 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 15:15:07.615481 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":317,"threads_started":1}
I20260926 15:15:07.635666 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:07.621518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:59967 (local address 127.6.209.254:37989)
0926 15:15:07.621725 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:07.621733 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:07.622403 (+ 670us) server_negotiation.cc:407] Connection header received
0926 15:15:07.622591 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:07.622598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:07.622649 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:07.622737 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:07.624004 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:07.624950 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:07.626703 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:07.627268 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:07.627400 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:07.629046 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:07.629053 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:07.629059 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:07.629099 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:07.632627 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:07.633412 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:07.633420 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:07.633425 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:07.633596 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:07.634194 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:07.634201 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:07.634206 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:07.634583 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:07.634747 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:07.635170 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 15:15:07.635347 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 15:15:07.635775 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:07.621645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:37989 (local address 127.6.209.193:59967)
0926 15:15:07.622302 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:07.622316 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:07.622465 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:07.622756 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:07.622760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:07.623472 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:07.623793 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:07.623840 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:07.625108 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:07.625113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:07.626572 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:07.626589 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:07.627460 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:07.627473 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:07.627649 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:07.628158 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:07.628172 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:07.628873 (+ 701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:07.632757 (+ 3884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:07.632761 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:07.632765 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:07.633296 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:07.633779 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:07.633792 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:07.633799 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:07.634011 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:07.634728 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:07.634734 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:07.634913 (+ 179us) client_negotiation.cc:769] Sending connection context
0926 15:15:07.635088 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 15:15:07.635367 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":375,"threads_started":1}
I20260926 15:15:07.645519 8788 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:40299
I20260926 15:15:07.645550 8922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:40299 every 8 connection(s)
I20260926 15:15:07.646713 8788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:15:07.648362 8923 heartbeater.cc:347] Connected to a master server at 127.6.209.254:37989
I20260926 15:15:07.648602 8923 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:07.649205 8923 heartbeater.cc:511] Master 127.6.209.254:37989 requested a full tablet report, sending...
I20260926 15:15:07.650815 8717 ts_manager.cc:194] Registered new tserver with Master: 9caa979a9710444182e0a168d483940e (127.6.209.193:40299)
I20260926 15:15:07.651345 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 8788
I20260926 15:15:07.651515 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:15:07.652294 6983 external_mini_cluster.cc:1428] Setting key af6991e2f65cfd9b4a3d7a44697fbd86
I20260926 15:15:07.652390 8717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.193@KRBTEST.COM'} at 127.6.209.193:33277
WARNING: no policy specified for kudu/127.6.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
I20260926 15:15:07.687887 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:37989
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:07.965078 8927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:07.965319 8927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:07.965469 8927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:07.971339 8927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:07.971457 8927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:07.971613 8927 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 15:15:07.971688 8927 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:15:07.971773 8927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:15:07.980165 8927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.209.254:37989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:07.981141 8927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:07.982250 8927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:07.993913 8932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:07.993847 8935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:07.993852 8933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:07.994498 8927 server_base.cc:1034] running on GCE node
I20260926 15:15:07.995014 8927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:07.995757 8927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:07.996941 8927 hybrid_clock.cc:648] HybridClock initialized: now 1790435707996914 us; error 38 us; skew 500 ppm
I20260926 15:15:07.997212 8927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:07 dist-test-slave-j6q7 krb5kdc[8672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435707, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:08.000344 8927 init.cc:377] Logged in from keytab as kudu/127.6.209.194@KRBTEST.COM (short username kudu)
I20260926 15:15:08.001750 8927 webserver.cc:460] Webserver started at http://127.6.209.194:40025/ using document root <none> and password file <none>
I20260926 15:15:08.002362 8927 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:08.002522 8927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:08.002781 8927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:08.005359 8927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "2cd15daf6de446359f72ef4769e57039"
format_stamp: "Formatted at 2026-09-26 15:15:07 on dist-test-slave-j6q7"
server_key: "0fb73962e48d59303ce1d0a315f4f54e"
server_key_iv: "bf8c20ecec5dae5d80661c926eb580b6"
server_key_version: "encryptionkey@0"
I20260926 15:15:08.005988 8927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "2cd15daf6de446359f72ef4769e57039"
format_stamp: "Formatted at 2026-09-26 15:15:07 on dist-test-slave-j6q7"
server_key: "0fb73962e48d59303ce1d0a315f4f54e"
server_key_iv: "bf8c20ecec5dae5d80661c926eb580b6"
server_key_version: "encryptionkey@0"
I20260926 15:15:08.010140 8927 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:15:08.013089 8942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:08.014246 8927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:15:08.014427 8927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "2cd15daf6de446359f72ef4769e57039"
format_stamp: "Formatted at 2026-09-26 15:15:07 on dist-test-slave-j6q7"
server_key: "0fb73962e48d59303ce1d0a315f4f54e"
server_key_iv: "bf8c20ecec5dae5d80661c926eb580b6"
server_key_version: "encryptionkey@0"
I20260926 15:15:08.014601 8927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:08.024699 8927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:08.027683 8927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:08.027923 8927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:08.030260 8927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:08.030355 8927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:08.030469 8927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:08.030532 8927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 15:15:08 dist-test-slave-j6q7 krb5kdc[8672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435707, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:08.051074 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:08.031144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:33265 (local address 127.6.209.254:37989)
0926 15:15:08.031280 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:08.031286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:08.031905 (+ 619us) server_negotiation.cc:407] Connection header received
0926 15:15:08.033700 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:08.033706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:08.033785 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:08.033902 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:08.035822 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.036921 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.038898 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.039621 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.039753 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:08.044483 (+ 4730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:08.044487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:08.044489 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:08.044516 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:08.047395 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:08.048382 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:08.048386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:08.048389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:08.048528 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:08.049099 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:08.049103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:08.049107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:08.049458 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:08.049607 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:08.050372 (+ 765us) server_negotiation.cc:299] Negotiation successful
0926 15:15:08.050639 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 15:15:08.051489 8951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:08.031131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:37989 (local address 127.6.209.194:33265)
0926 15:15:08.031778 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:08.031809 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:08.033445 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:08.034140 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:08.034153 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:08.034823 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:08.035652 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:08.035681 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.037075 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.037080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:08.038696 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:08.038709 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.039841 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.039846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:08.039981 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:08.040572 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:08.040606 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:08.044260 (+ 3654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:08.047530 (+ 3270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:08.047540 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:08.047556 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:08.048223 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:08.048660 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:08.048667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:08.048675 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:08.048945 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:08.049628 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:08.049635 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:08.049936 (+ 301us) client_negotiation.cc:769] Sending connection context
0926 15:15:08.050253 (+ 317us) client_negotiation.cc:240] Negotiation successful
0926 15:15:08.050561 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":278,"threads_started":1}
I20260926 15:15:08.069295 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:08.056197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:45053 (local address 127.6.209.254:37989)
0926 15:15:08.056330 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:08.056334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:08.056957 (+ 623us) server_negotiation.cc:407] Connection header received
0926 15:15:08.057116 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:08.057123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:08.057205 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:08.057313 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:08.057958 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.059145 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.060249 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.061054 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.061206 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:08.063228 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:08.063236 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:08.063243 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:08.063305 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:08.066615 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:08.067503 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:08.067506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:08.067508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:08.067601 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:08.067993 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:08.067996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:08.067998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:08.068228 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:08.068321 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:08.068779 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 15:15:08.069006 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 15:15:08.069621 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:08.056301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:37989 (local address 127.6.209.194:45053)
0926 15:15:08.056849 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:08.056867 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:08.056992 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:08.057322 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:08.057325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:08.057629 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:08.057847 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:08.057853 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.059261 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.059264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:08.060132 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:08.060141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.061363 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.061368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:08.061486 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:08.062136 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:08.062151 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:08.063017 (+ 866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:08.066732 (+ 3715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:08.066737 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:08.066741 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:08.067379 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:08.067704 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:08.067710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:08.067715 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:08.067896 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:08.068329 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:08.068336 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:08.068547 (+ 211us) client_negotiation.cc:769] Sending connection context
0926 15:15:08.068703 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 15:15:08.069105 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":296,"threads_started":1}
I20260926 15:15:08.083940 8927 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:33079
I20260926 15:15:08.083979 9061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:33079 every 8 connection(s)
I20260926 15:15:08.085062 8927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:15:08.086612 9062 heartbeater.cc:347] Connected to a master server at 127.6.209.254:37989
I20260926 15:15:08.086798 9062 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:08.087286 9062 heartbeater.cc:511] Master 127.6.209.254:37989 requested a full tablet report, sending...
I20260926 15:15:08.088311 8716 ts_manager.cc:194] Registered new tserver with Master: 2cd15daf6de446359f72ef4769e57039 (127.6.209.194:33079)
I20260926 15:15:08.089120 8716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.194@KRBTEST.COM'} at 127.6.209.194:33265
I20260926 15:15:08.092605 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 8927
I20260926 15:15:08.092741 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:15:08.093207 6983 external_mini_cluster.cc:1428] Setting key 259d1348cea7731a16cbfa893fdedf64
WARNING: no policy specified for kudu/127.6.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
I20260926 15:15:08.129427 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:37989
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:08.406355 9066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:08.406569 9066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:08.406694 9066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:08.413416 9066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:08.413532 9066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:08.413681 9066 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 15:15:08.413756 9066 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:15:08.413874 9066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:15:08.423013 9066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.209.254:37989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:08.424047 9066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:08.425390 9066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:08.437489 9071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:08.437558 9074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:08.437577 9072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:08.438306 9066 server_base.cc:1034] running on GCE node
I20260926 15:15:08.438792 9066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:08.439565 9066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:08.440768 9066 hybrid_clock.cc:648] HybridClock initialized: now 1790435708440728 us; error 51 us; skew 500 ppm
I20260926 15:15:08.441052 9066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:08 dist-test-slave-j6q7 krb5kdc[8672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435708, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:08.443985 9066 init.cc:377] Logged in from keytab as kudu/127.6.209.195@KRBTEST.COM (short username kudu)
I20260926 15:15:08.445297 9066 webserver.cc:460] Webserver started at http://127.6.209.195:37677/ using document root <none> and password file <none>
I20260926 15:15:08.445850 9066 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:08.445977 9066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:08.446252 9066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:08.448529 9066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f"
format_stamp: "Formatted at 2026-09-26 15:15:08 on dist-test-slave-j6q7"
server_key: "ab8b3701a35974e560e5db6bd24e555b"
server_key_iv: "5bf8ff955ef411af21072a9cc0e5a4da"
server_key_version: "encryptionkey@0"
I20260926 15:15:08.449199 9066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f"
format_stamp: "Formatted at 2026-09-26 15:15:08 on dist-test-slave-j6q7"
server_key: "ab8b3701a35974e560e5db6bd24e555b"
server_key_iv: "5bf8ff955ef411af21072a9cc0e5a4da"
server_key_version: "encryptionkey@0"
I20260926 15:15:08.453959 9066 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:15:08.456888 9081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:08.457940 9066 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:15:08.458101 9066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f"
format_stamp: "Formatted at 2026-09-26 15:15:08 on dist-test-slave-j6q7"
server_key: "ab8b3701a35974e560e5db6bd24e555b"
server_key_iv: "5bf8ff955ef411af21072a9cc0e5a4da"
server_key_version: "encryptionkey@0"
I20260926 15:15:08.458267 9066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:08.494285 9066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:08.497498 9066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:08.497764 9066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:08.500079 9066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:08.500205 9066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:08.500309 9066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:08.500383 9066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 15:15:08 dist-test-slave-j6q7 krb5kdc[8672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435708, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:08.524221 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:08.501612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:37989 (local address 127.6.209.195:37785)
0926 15:15:08.502365 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:08.502415 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:08.504453 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:08.505551 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:08.505562 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:08.506348 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:08.507151 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:08.507179 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.508688 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.508710 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:08.510383 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:08.510397 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.511335 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.511342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:08.511587 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:08.512280 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:08.512302 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:08.515481 (+ 3179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:08.519093 (+ 3612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:08.519106 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:08.519125 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:08.519897 (+ 772us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:08.520398 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:08.520404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:08.520412 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:08.520663 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:08.521319 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:08.521327 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:08.521801 (+ 474us) client_negotiation.cc:769] Sending connection context
0926 15:15:08.522142 (+ 341us) client_negotiation.cc:240] Negotiation successful
0926 15:15:08.522640 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":339,"threads_started":1}
I20260926 15:15:08.524878 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:08.500952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:37785 (local address 127.6.209.254:37989)
0926 15:15:08.501173 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:08.501178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:08.502531 (+ 1353us) server_negotiation.cc:407] Connection header received
0926 15:15:08.504822 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:08.504839 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:08.504941 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:08.505025 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:08.507349 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.508501 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.510558 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.511109 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.511236 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:08.515674 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:08.515682 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:08.515688 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:08.515733 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:08.518957 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:08.520078 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:08.520085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:08.520092 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:08.520249 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:08.520776 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:08.520783 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:08.520787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:08.521200 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:08.521339 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:08.524301 (+ 2962us) server_negotiation.cc:299] Negotiation successful
0926 15:15:08.524571 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 15:15:08.538255 9066 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:35275
I20260926 15:15:08.538285 9200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:35275 every 8 connection(s)
I20260926 15:15:08.539521 9066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:15:08.541505 9201 heartbeater.cc:347] Connected to a master server at 127.6.209.254:37989
I20260926 15:15:08.541715 9201 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:08.542472 9201 heartbeater.cc:511] Master 127.6.209.254:37989 requested a full tablet report, sending...
I20260926 15:15:08.542645 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 9066
I20260926 15:15:08.542790 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:15:08.543310 6983 external_mini_cluster.cc:1428] Setting key 81a11d2b89735ecf4acff141f8647f71
I20260926 15:15:08.543874 8716 ts_manager.cc:194] Registered new tserver with Master: c2f9aa3153fc487c8c5cdb06e521ba8f (127.6.209.195:35275)
I20260926 15:15:08.544778 9182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:08.530992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:37989 (local address 127.6.209.195:41597)
0926 15:15:08.531559 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:08.531576 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:08.531732 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:08.532135 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:08.532140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:08.532786 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:08.533355 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:08.533374 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.534641 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.534646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:08.536080 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:08.536090 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.536860 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.536870 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:08.536995 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:08.537578 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:08.537602 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:08.538952 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:08.542133 (+ 3181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:08.542136 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:08.542139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:08.542477 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:08.543105 (+ 628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:08.543109 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:08.543112 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:08.543238 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:08.543836 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:08.543844 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:08.544029 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 15:15:08.544146 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 15:15:08.544405 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":330,"threads_started":1}
I20260926 15:15:08.544927 8716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.195@KRBTEST.COM'} at 127.6.209.195:37785
I20260926 15:15:08.545635 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:08.530842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:41597 (local address 127.6.209.254:37989)
0926 15:15:08.531002 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:08.531005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:08.531654 (+ 649us) server_negotiation.cc:407] Connection header received
0926 15:15:08.531869 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:08.531874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:08.531987 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:08.532079 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:08.533597 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.534501 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.536234 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.536653 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.536767 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:08.539105 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:08.539109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:08.539112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:08.539136 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:08.542017 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:08.542601 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:08.542607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:08.542610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:08.542745 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:08.543344 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:08.543350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:08.543353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:08.543701 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:08.544990 (+ 1289us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:08.545124 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 15:15:08.545332 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 15:15:08.545964 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 15:15:08 dist-test-slave-j6q7 krb5kdc[8672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 15:15:08 dist-test-slave-j6q7 krb5kdc[8672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:08.587289 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:08.575209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33472 (local address 127.6.209.254:37989)
0926 15:15:08.575355 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:08.575359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:08.575734 (+ 375us) server_negotiation.cc:407] Connection header received
0926 15:15:08.575860 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:08.575864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:08.575922 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:08.576001 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:08.577348 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.578005 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.579019 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.579361 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.579438 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:08.582816 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:08.582820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:08.582823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:08.582849 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:08.585051 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:08.585774 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:08.585779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:08.585782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:08.585936 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:08.586282 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:08.586285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:08.586286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:08.586544 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:08.586699 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:08.586868 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 15:15:08.586996 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:15:08.590576 8717 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:15:08.596524 8717 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:15:08.618688 9219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:08.608400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57308 (local address 127.6.209.193:40299)
0926 15:15:08.608988 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:08.608998 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:08.612039 (+ 3041us) server_negotiation.cc:407] Connection header received
0926 15:15:08.612142 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:08.612147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:08.612334 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:08.612489 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:08.613235 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.614641 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.616513 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.617495 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.617624 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:08.617792 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:08.618213 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 15:15:08.618393 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":261,"threads_started":1}
I20260926 15:15:08.619446 9217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:08.607849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36248 (local address 127.6.209.194:33079)
0926 15:15:08.608326 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:08.608338 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:08.611589 (+ 3251us) server_negotiation.cc:407] Connection header received
0926 15:15:08.612284 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:08.612290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:08.612484 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:08.612694 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:08.613433 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.614768 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.617277 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.618501 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.618627 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:08.618796 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:08.619009 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 15:15:08.619167 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":241,"threads_started":1}
I20260926 15:15:08.619658 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:08.607873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:40299 (local address 127.0.0.1:57308)
0926 15:15:08.611937 (+ 4064us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:08.611949 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:08.612045 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:08.612556 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:08.612560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:08.612584 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:08.613051 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:08.613064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.614744 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.614747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:08.616374 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:08.616384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.617676 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.617682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:08.617926 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:08.617973 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 15:15:08.618937 (+ 964us) client_negotiation.cc:240] Negotiation successful
0926 15:15:08.619191 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3977,"mutex_wait_us":89,"thread_start_us":3277,"threads_started":1}
I20260926 15:15:08.619681 9218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:08.607382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:33079 (local address 127.0.0.1:36248)
0926 15:15:08.611468 (+ 4086us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:08.611488 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:08.612129 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:08.612886 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:08.612890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:08.612921 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:08.613242 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:08.613251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.614884 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.614890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:08.617128 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:08.617136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.618682 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.618689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:08.618830 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:08.618859 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 15:15:08.618937 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 15:15:08.619191 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3932,"mutex_wait_us":123,"thread_start_us":4379,"threads_started":1}
I20260926 15:15:08.620550 9214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:08.606795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:35275 (local address 127.0.0.1:40046)
0926 15:15:08.607549 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:08.607586 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:08.607733 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:08.612311 (+ 4578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:08.612316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:08.612339 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:08.612725 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:08.612741 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.614682 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.614689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:08.617114 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:08.617126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.619717 (+ 2591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.619722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:08.619951 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:08.619985 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 15:15:08.620087 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 15:15:08.620210 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":150,"thread_start_us":418,"threads_started":1}
I20260926 15:15:08.622603 8992 tablet_service.cc:1467] Processing CreateTablet for tablet eb943a8392114ce5ab49b0f8255f5f58 (DEFAULT_TABLE table=test-table [id=931c602e898b4e41ad3f10cc7e005d1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:08.622679 8857 tablet_service.cc:1467] Processing CreateTablet for tablet eb943a8392114ce5ab49b0f8255f5f58 (DEFAULT_TABLE table=test-table [id=931c602e898b4e41ad3f10cc7e005d1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:08.623701 8992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb943a8392114ce5ab49b0f8255f5f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:08.623761 8857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb943a8392114ce5ab49b0f8255f5f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:08.625571 9215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:08.607104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40046 (local address 127.6.209.195:35275)
0926 15:15:08.611796 (+ 4692us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:08.611801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:08.611823 (+ 22us) server_negotiation.cc:407] Connection header received
0926 15:15:08.611887 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:08.611891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:08.612098 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:08.612295 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:08.612930 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.614566 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.617877 (+ 3311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.619324 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.620830 (+ 1506us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:08.625075 (+ 4245us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:08.625137 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 15:15:08.625275 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4577,"thread_start_us":4422,"threads_started":1}
I20260926 15:15:08.628471 9131 tablet_service.cc:1467] Processing CreateTablet for tablet eb943a8392114ce5ab49b0f8255f5f58 (DEFAULT_TABLE table=test-table [id=931c602e898b4e41ad3f10cc7e005d1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:08.629670 9131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb943a8392114ce5ab49b0f8255f5f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:08.634160 9220 tablet_bootstrap.cc:492] T eb943a8392114ce5ab49b0f8255f5f58 P 2cd15daf6de446359f72ef4769e57039: Bootstrap starting.
I20260926 15:15:08.636276 9221 tablet_bootstrap.cc:492] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e: Bootstrap starting.
I20260926 15:15:08.638029 9220 tablet_bootstrap.cc:654] T eb943a8392114ce5ab49b0f8255f5f58 P 2cd15daf6de446359f72ef4769e57039: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:08.639096 9221 tablet_bootstrap.cc:654] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:08.639165 9220 log.cc:824] T eb943a8392114ce5ab49b0f8255f5f58 P 2cd15daf6de446359f72ef4769e57039: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:08.640302 9221 log.cc:824] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:08.647730 9224 tablet_bootstrap.cc:492] T eb943a8392114ce5ab49b0f8255f5f58 P c2f9aa3153fc487c8c5cdb06e521ba8f: Bootstrap starting.
I20260926 15:15:08.648923 9220 tablet_bootstrap.cc:492] T eb943a8392114ce5ab49b0f8255f5f58 P 2cd15daf6de446359f72ef4769e57039: No bootstrap required, opened a new log
I20260926 15:15:08.649147 9220 ts_tablet_manager.cc:1397] T eb943a8392114ce5ab49b0f8255f5f58 P 2cd15daf6de446359f72ef4769e57039: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20260926 15:15:08.650836 9224 tablet_bootstrap.cc:654] T eb943a8392114ce5ab49b0f8255f5f58 P c2f9aa3153fc487c8c5cdb06e521ba8f: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:08.651624 9221 tablet_bootstrap.cc:492] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e: No bootstrap required, opened a new log
I20260926 15:15:08.651818 9224 log.cc:824] T eb943a8392114ce5ab49b0f8255f5f58 P c2f9aa3153fc487c8c5cdb06e521ba8f: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:08.652072 9221 ts_tablet_manager.cc:1397] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20260926 15:15:08.655453 8923 heartbeater.cc:503] Master 127.6.209.254:37989 was elected leader, sending a full tablet report...
I20260926 15:15:08.655592 9224 tablet_bootstrap.cc:492] T eb943a8392114ce5ab49b0f8255f5f58 P c2f9aa3153fc487c8c5cdb06e521ba8f: No bootstrap required, opened a new log
I20260926 15:15:08.655803 9224 ts_tablet_manager.cc:1397] T eb943a8392114ce5ab49b0f8255f5f58 P c2f9aa3153fc487c8c5cdb06e521ba8f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 15:15:08.659075 9220 raft_consensus.cc:348] T eb943a8392114ce5ab49b0f8255f5f58 P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } } peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } }
I20260926 15:15:08.659422 9220 raft_consensus.cc:374] T eb943a8392114ce5ab49b0f8255f5f58 P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:08.659498 9220 raft_consensus.cc:729] T eb943a8392114ce5ab49b0f8255f5f58 P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd15daf6de446359f72ef4769e57039, State: Initialized, Role: FOLLOWER
I20260926 15:15:08.659979 9220 consensus_queue.cc:260] T eb943a8392114ce5ab49b0f8255f5f58 P 2cd15daf6de446359f72ef4769e57039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } } peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } }
I20260926 15:15:08.660916 9062 heartbeater.cc:503] Master 127.6.209.254:37989 was elected leader, sending a full tablet report...
I20260926 15:15:08.661618 9220 ts_tablet_manager.cc:1428] T eb943a8392114ce5ab49b0f8255f5f58 P 2cd15daf6de446359f72ef4769e57039: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260926 15:15:08.664034 9221 raft_consensus.cc:348] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } } peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } }
I20260926 15:15:08.664389 9221 raft_consensus.cc:374] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:08.664507 9221 raft_consensus.cc:729] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9caa979a9710444182e0a168d483940e, State: Initialized, Role: FOLLOWER
I20260926 15:15:08.664696 9224 raft_consensus.cc:348] T eb943a8392114ce5ab49b0f8255f5f58 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } } peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } }
I20260926 15:15:08.665096 9224 raft_consensus.cc:374] T eb943a8392114ce5ab49b0f8255f5f58 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:08.665179 9224 raft_consensus.cc:729] T eb943a8392114ce5ab49b0f8255f5f58 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2f9aa3153fc487c8c5cdb06e521ba8f, State: Initialized, Role: FOLLOWER
I20260926 15:15:08.665066 9221 consensus_queue.cc:260] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } } peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } }
I20260926 15:15:08.665670 9224 consensus_queue.cc:260] T eb943a8392114ce5ab49b0f8255f5f58 P c2f9aa3153fc487c8c5cdb06e521ba8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } } peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } }
I20260926 15:15:08.666604 9201 heartbeater.cc:503] Master 127.6.209.254:37989 was elected leader, sending a full tablet report...
I20260926 15:15:08.667150 9224 ts_tablet_manager.cc:1428] T eb943a8392114ce5ab49b0f8255f5f58 P c2f9aa3153fc487c8c5cdb06e521ba8f: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260926 15:15:08.669917 9221 ts_tablet_manager.cc:1428] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e: Time spent starting tablet: real 0.018s user 0.010s sys 0.004s
W20260926 15:15:08.790958 9202 tablet.cc:2369] T eb943a8392114ce5ab49b0f8255f5f58 P c2f9aa3153fc487c8c5cdb06e521ba8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:15:08.814873 9227 raft_consensus.cc:482] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:15:08.815181 9227 raft_consensus.cc:504] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } } peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } }
I20260926 15:15:08.816668 9227 leader_election.cc:288] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2f9aa3153fc487c8c5cdb06e521ba8f (127.6.209.195:35275), 2cd15daf6de446359f72ef4769e57039 (127.6.209.194:33079)
I20260926 15:15:08.822414 9215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:08.817114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:58785 (local address 127.6.209.195:35275)
0926 15:15:08.817315 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:08.817321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:08.817637 (+ 316us) server_negotiation.cc:407] Connection header received
0926 15:15:08.817851 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:08.817858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:08.817923 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:08.818005 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:08.818538 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.819259 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.820885 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.821576 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.821743 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:08.821807 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:08.822082 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 15:15:08.822186 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 15:15:08.822638 9217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:08.817013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:54317 (local address 127.6.209.194:33079)
0926 15:15:08.817217 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:08.817225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:08.818179 (+ 954us) server_negotiation.cc:407] Connection header received
0926 15:15:08.818386 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:08.818394 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:08.818442 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:08.818532 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:08.819000 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.819599 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.821176 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.821836 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.822012 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:08.822075 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:08.822317 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 15:15:08.822420 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 15:15:08.822687 9230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:08.817479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:33079 (local address 127.6.209.193:54317)
0926 15:15:08.818077 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:08.818093 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:08.818264 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:08.818537 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:08.818541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:08.818559 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:08.818864 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:08.818873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.819726 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.819731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:08.821051 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:08.821061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.822037 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.822043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:08.822173 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:08.822191 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 15:15:08.822295 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 15:15:08.822403 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":317,"threads_started":1}
I20260926 15:15:08.822739 9229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:08.816946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:35275 (local address 127.6.209.193:58785)
0926 15:15:08.817533 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:08.817552 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:08.817704 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:08.818018 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:08.818022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:08.818050 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:08.818415 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:08.818423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.819364 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.819371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:08.820736 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:08.820746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.821755 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.821760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:08.821920 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:08.821941 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 15:15:08.822059 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 15:15:08.822168 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":1,"thread_start_us":286,"threads_started":1}
I20260926 15:15:08.823599 9017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb943a8392114ce5ab49b0f8255f5f58" candidate_uuid: "9caa979a9710444182e0a168d483940e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd15daf6de446359f72ef4769e57039" is_pre_election: true
I20260926 15:15:08.823598 9151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb943a8392114ce5ab49b0f8255f5f58" candidate_uuid: "9caa979a9710444182e0a168d483940e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" is_pre_election: true
I20260926 15:15:08.824059 9017 raft_consensus.cc:2454] T eb943a8392114ce5ab49b0f8255f5f58 P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9caa979a9710444182e0a168d483940e in term 0.
I20260926 15:15:08.824075 9151 raft_consensus.cc:2454] T eb943a8392114ce5ab49b0f8255f5f58 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9caa979a9710444182e0a168d483940e in term 0.
I20260926 15:15:08.824640 8804 leader_election.cc:302] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cd15daf6de446359f72ef4769e57039, 9caa979a9710444182e0a168d483940e; no voters:
I20260926 15:15:08.824985 9227 raft_consensus.cc:2781] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:15:08.825112 9227 raft_consensus.cc:482] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:15:08.825202 9227 raft_consensus.cc:3037] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:08.827322 9227 raft_consensus.cc:504] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } } peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } }
I20260926 15:15:08.827834 9227 leader_election.cc:288] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e [CANDIDATE]: Term 1 election: Requested vote from peers c2f9aa3153fc487c8c5cdb06e521ba8f (127.6.209.195:35275), 2cd15daf6de446359f72ef4769e57039 (127.6.209.194:33079)
I20260926 15:15:08.828091 9151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb943a8392114ce5ab49b0f8255f5f58" candidate_uuid: "9caa979a9710444182e0a168d483940e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f"
I20260926 15:15:08.828249 9151 raft_consensus.cc:3037] T eb943a8392114ce5ab49b0f8255f5f58 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:08.828236 9017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb943a8392114ce5ab49b0f8255f5f58" candidate_uuid: "9caa979a9710444182e0a168d483940e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd15daf6de446359f72ef4769e57039"
I20260926 15:15:08.828393 9017 raft_consensus.cc:3037] T eb943a8392114ce5ab49b0f8255f5f58 P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:08.830199 9151 raft_consensus.cc:2454] T eb943a8392114ce5ab49b0f8255f5f58 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9caa979a9710444182e0a168d483940e in term 1.
I20260926 15:15:08.830353 9017 raft_consensus.cc:2454] T eb943a8392114ce5ab49b0f8255f5f58 P 2cd15daf6de446359f72ef4769e57039 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9caa979a9710444182e0a168d483940e in term 1.
I20260926 15:15:08.830530 8807 leader_election.cc:302] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9caa979a9710444182e0a168d483940e, c2f9aa3153fc487c8c5cdb06e521ba8f; no voters:
I20260926 15:15:08.830753 9227 raft_consensus.cc:2781] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:08.831132 9227 raft_consensus.cc:686] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e [term 1 LEADER]: Becoming Leader. State: Replica: 9caa979a9710444182e0a168d483940e, State: Running, Role: LEADER
I20260926 15:15:08.831522 9227 consensus_queue.cc:237] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } } peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } }
I20260926 15:15:08.835741 8716 catalog_manager.cc:5495] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9caa979a9710444182e0a168d483940e (127.6.209.193). New cstate: current_term: 1 leader_uuid: "9caa979a9710444182e0a168d483940e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } health_report { overall_health: UNKNOWN } } }
W20260926 15:15:08.836690 9063 tablet.cc:2369] T eb943a8392114ce5ab49b0f8255f5f58 P 2cd15daf6de446359f72ef4769e57039: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 15:15:08 dist-test-slave-j6q7 krb5kdc[8672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435707, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:08.894726 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:08.882954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:37989 (local address 127.0.0.1:33480)
0926 15:15:08.883443 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:08.883458 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:08.883630 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:08.883923 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:08.883927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:08.884587 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:08.884826 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:08.884855 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.885817 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.885825 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:08.886708 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:08.886718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.887372 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.887377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:08.887546 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:08.888101 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:08.888116 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:08.890016 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:08.892216 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:08.892225 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:08.892230 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:08.892592 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:08.892900 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:08.892903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:08.892904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:08.893015 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:08.893525 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:08.893535 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:08.893751 (+ 216us) client_negotiation.cc:769] Sending connection context
0926 15:15:08.893954 (+ 203us) client_negotiation.cc:240] Negotiation successful
0926 15:15:08.894283 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":251,"threads_started":1}
I20260926 15:15:08.894796 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:08.882897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33480 (local address 127.6.209.254:37989)
0926 15:15:08.883042 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:08.883046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:08.883555 (+ 509us) server_negotiation.cc:407] Connection header received
0926 15:15:08.883746 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:08.883758 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:08.883807 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:08.883905 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:08.884994 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.885700 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.886835 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:08.887223 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:08.887340 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:08.890146 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:08.890150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:08.890153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:08.890176 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:08.892118 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:08.892674 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:08.892677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:08.892679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:08.892773 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:08.893093 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:08.893095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:08.893097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:08.893377 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:08.893512 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:08.894086 (+ 574us) server_negotiation.cc:299] Negotiation successful
0926 15:15:08.894429 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":51}
I20260926 15:15:08.898864 8716 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:33480:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 15:15:08.899778 8716 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 15:15:08.901095 8924 tablet.cc:2369] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:15:08.905926 9131 tablet_service.cc:1467] Processing CreateTablet for tablet 656d5a418fc04e19b7b93af652ebb2f6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b16437e94ef24440b8a24dde5517ba88]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:15:08.905763 8857 tablet_service.cc:1467] Processing CreateTablet for tablet 656d5a418fc04e19b7b93af652ebb2f6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b16437e94ef24440b8a24dde5517ba88]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:15:08.905891 8992 tablet_service.cc:1467] Processing CreateTablet for tablet 656d5a418fc04e19b7b93af652ebb2f6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b16437e94ef24440b8a24dde5517ba88]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:15:08.906638 9131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 656d5a418fc04e19b7b93af652ebb2f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:08.906648 8992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 656d5a418fc04e19b7b93af652ebb2f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:08.906647 8857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 656d5a418fc04e19b7b93af652ebb2f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:08.911168 9224 tablet_bootstrap.cc:492] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f: Bootstrap starting.
I20260926 15:15:08.912315 9224 tablet_bootstrap.cc:654] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:08.913089 9221 tablet_bootstrap.cc:492] T 656d5a418fc04e19b7b93af652ebb2f6 P 9caa979a9710444182e0a168d483940e: Bootstrap starting.
I20260926 15:15:08.913198 9220 tablet_bootstrap.cc:492] T 656d5a418fc04e19b7b93af652ebb2f6 P 2cd15daf6de446359f72ef4769e57039: Bootstrap starting.
I20260926 15:15:08.914654 9221 tablet_bootstrap.cc:654] T 656d5a418fc04e19b7b93af652ebb2f6 P 9caa979a9710444182e0a168d483940e: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:08.914716 9220 tablet_bootstrap.cc:654] T 656d5a418fc04e19b7b93af652ebb2f6 P 2cd15daf6de446359f72ef4769e57039: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:08.916702 9221 tablet_bootstrap.cc:492] T 656d5a418fc04e19b7b93af652ebb2f6 P 9caa979a9710444182e0a168d483940e: No bootstrap required, opened a new log
I20260926 15:15:08.916712 9224 tablet_bootstrap.cc:492] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f: No bootstrap required, opened a new log
I20260926 15:15:08.916855 9221 ts_tablet_manager.cc:1397] T 656d5a418fc04e19b7b93af652ebb2f6 P 9caa979a9710444182e0a168d483940e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 15:15:08.916883 9224 ts_tablet_manager.cc:1397] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 15:15:08.917578 9224 raft_consensus.cc:348] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:08.917573 9221 raft_consensus.cc:348] T 656d5a418fc04e19b7b93af652ebb2f6 P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:08.917780 9220 tablet_bootstrap.cc:492] T 656d5a418fc04e19b7b93af652ebb2f6 P 2cd15daf6de446359f72ef4769e57039: No bootstrap required, opened a new log
I20260926 15:15:08.917791 9224 raft_consensus.cc:374] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:08.917809 9221 raft_consensus.cc:374] T 656d5a418fc04e19b7b93af652ebb2f6 P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:08.917869 9224 raft_consensus.cc:729] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2f9aa3153fc487c8c5cdb06e521ba8f, State: Initialized, Role: FOLLOWER
I20260926 15:15:08.917915 9220 ts_tablet_manager.cc:1397] T 656d5a418fc04e19b7b93af652ebb2f6 P 2cd15daf6de446359f72ef4769e57039: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 15:15:08.917896 9221 raft_consensus.cc:729] T 656d5a418fc04e19b7b93af652ebb2f6 P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9caa979a9710444182e0a168d483940e, State: Initialized, Role: FOLLOWER
I20260926 15:15:08.918103 9224 consensus_queue.cc:260] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:08.918111 9221 consensus_queue.cc:260] T 656d5a418fc04e19b7b93af652ebb2f6 P 9caa979a9710444182e0a168d483940e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:08.918583 9224 ts_tablet_manager.cc:1428] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:15:08.918587 9221 ts_tablet_manager.cc:1428] T 656d5a418fc04e19b7b93af652ebb2f6 P 9caa979a9710444182e0a168d483940e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260926 15:15:08.918689 9220 raft_consensus.cc:348] T 656d5a418fc04e19b7b93af652ebb2f6 P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:08.918912 9220 raft_consensus.cc:374] T 656d5a418fc04e19b7b93af652ebb2f6 P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:08.919010 9220 raft_consensus.cc:729] T 656d5a418fc04e19b7b93af652ebb2f6 P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd15daf6de446359f72ef4769e57039, State: Initialized, Role: FOLLOWER
I20260926 15:15:08.919018 9228 tablet_replica.cc:439] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } } }
I20260926 15:15:08.919371 9228 tablet_replica.cc:442] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:15:08.919281 9220 consensus_queue.cc:260] T 656d5a418fc04e19b7b93af652ebb2f6 P 2cd15daf6de446359f72ef4769e57039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:08.919610 9227 tablet_replica.cc:439] T 656d5a418fc04e19b7b93af652ebb2f6 P 9caa979a9710444182e0a168d483940e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } } }
I20260926 15:15:08.919845 9227 tablet_replica.cc:442] T 656d5a418fc04e19b7b93af652ebb2f6 P 9caa979a9710444182e0a168d483940e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:15:08.919878 9226 tablet_replica.cc:439] T 656d5a418fc04e19b7b93af652ebb2f6 P 2cd15daf6de446359f72ef4769e57039: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } } }
I20260926 15:15:08.920163 9226 tablet_replica.cc:442] T 656d5a418fc04e19b7b93af652ebb2f6 P 2cd15daf6de446359f72ef4769e57039: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:15:08.920564 9220 ts_tablet_manager.cc:1428] T 656d5a418fc04e19b7b93af652ebb2f6 P 2cd15daf6de446359f72ef4769e57039: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 15:15:09.110831 9228 raft_consensus.cc:482] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:15:09.110996 9228 raft_consensus.cc:504] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:09.112207 9228 leader_election.cc:288] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9caa979a9710444182e0a168d483940e (127.6.209.193:40299), 2cd15daf6de446359f72ef4769e57039 (127.6.209.194:33079)
I20260926 15:15:09.119232 9217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:09.112634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:43213 (local address 127.6.209.194:33079)
0926 15:15:09.112950 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:09.112958 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:09.113036 (+ 78us) server_negotiation.cc:407] Connection header received
0926 15:15:09.113230 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:09.113239 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:09.113320 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:09.113429 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:09.113858 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:09.114952 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:09.117207 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:09.118287 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:09.118437 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:09.118500 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:09.118852 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 15:15:09.118963 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218}
I20260926 15:15:09.119222 9219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:09.112581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:38711 (local address 127.6.209.193:40299)
0926 15:15:09.112981 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:09.112991 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:09.113308 (+ 317us) server_negotiation.cc:407] Connection header received
0926 15:15:09.113436 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:09.113441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:09.113504 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:09.113595 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:09.114078 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:09.115091 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:09.117204 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:09.118286 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:09.118447 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:09.118500 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:09.118852 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 15:15:09.118963 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260926 15:15:09.119415 9242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:09.112455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:33079 (local address 127.6.209.195:43213)
0926 15:15:09.112941 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:09.112960 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:09.113102 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:09.113468 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:09.113471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:09.113489 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:09.113718 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:09.113732 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:09.115069 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:09.115072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:09.117079 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:09.117088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:09.118538 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:09.118543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:09.118725 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:09.118745 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 15:15:09.118820 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 15:15:09.119024 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":95,"thread_start_us":252,"threads_started":1}
I20260926 15:15:09.119469 9243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:09.112455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:40299 (local address 127.6.209.195:38711)
0926 15:15:09.113214 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:09.113226 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:09.113316 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:09.113618 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:09.113623 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:09.113652 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:09.113941 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:09.113949 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:09.115202 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:09.115206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:09.117065 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:09.117079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:09.118453 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:09.118461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:09.118684 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:09.118733 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 15:15:09.118820 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 15:15:09.118996 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":159,"thread_start_us":501,"threads_started":1}
I20260926 15:15:09.120041 9017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "656d5a418fc04e19b7b93af652ebb2f6" candidate_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd15daf6de446359f72ef4769e57039" is_pre_election: true
I20260926 15:15:09.120306 9017 raft_consensus.cc:2454] T 656d5a418fc04e19b7b93af652ebb2f6 P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2f9aa3153fc487c8c5cdb06e521ba8f in term 0.
I20260926 15:15:09.120484 8878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "656d5a418fc04e19b7b93af652ebb2f6" candidate_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9caa979a9710444182e0a168d483940e" is_pre_election: true
I20260926 15:15:09.120694 9082 leader_election.cc:302] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cd15daf6de446359f72ef4769e57039, c2f9aa3153fc487c8c5cdb06e521ba8f; no voters:
I20260926 15:15:09.120874 8878 raft_consensus.cc:2454] T 656d5a418fc04e19b7b93af652ebb2f6 P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2f9aa3153fc487c8c5cdb06e521ba8f in term 0.
I20260926 15:15:09.121176 9228 raft_consensus.cc:2781] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:15:09.121322 9228 raft_consensus.cc:482] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:15:09.121447 9228 raft_consensus.cc:3037] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:09.123409 9228 raft_consensus.cc:504] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:09.123942 9228 leader_election.cc:288] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f [CANDIDATE]: Term 1 election: Requested vote from peers 9caa979a9710444182e0a168d483940e (127.6.209.193:40299), 2cd15daf6de446359f72ef4769e57039 (127.6.209.194:33079)
I20260926 15:15:09.124307 9017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "656d5a418fc04e19b7b93af652ebb2f6" candidate_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd15daf6de446359f72ef4769e57039"
I20260926 15:15:09.124398 8878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "656d5a418fc04e19b7b93af652ebb2f6" candidate_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9caa979a9710444182e0a168d483940e"
I20260926 15:15:09.124521 9017 raft_consensus.cc:3037] T 656d5a418fc04e19b7b93af652ebb2f6 P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:09.124635 8878 raft_consensus.cc:3037] T 656d5a418fc04e19b7b93af652ebb2f6 P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:09.126286 9017 raft_consensus.cc:2454] T 656d5a418fc04e19b7b93af652ebb2f6 P 2cd15daf6de446359f72ef4769e57039 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2f9aa3153fc487c8c5cdb06e521ba8f in term 1.
I20260926 15:15:09.126600 8878 raft_consensus.cc:2454] T 656d5a418fc04e19b7b93af652ebb2f6 P 9caa979a9710444182e0a168d483940e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2f9aa3153fc487c8c5cdb06e521ba8f in term 1.
I20260926 15:15:09.126619 9082 leader_election.cc:302] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cd15daf6de446359f72ef4769e57039, c2f9aa3153fc487c8c5cdb06e521ba8f; no voters:
I20260926 15:15:09.126982 9228 raft_consensus.cc:2781] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:09.127394 9228 raft_consensus.cc:686] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 1 LEADER]: Becoming Leader. State: Replica: c2f9aa3153fc487c8c5cdb06e521ba8f, State: Running, Role: LEADER
I20260926 15:15:09.127861 9228 consensus_queue.cc:237] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:09.130975 9244 tablet_replica.cc:439] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f: TxnStatusTablet state changed. Reason: New leader c2f9aa3153fc487c8c5cdb06e521ba8f. Latest consensus state: current_term: 1 leader_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } } }
I20260926 15:15:09.131253 9244 tablet_replica.cc:442] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:15:09.131826 9246 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 15:15:09.132041 9246 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 15:15:09.132277 8715 catalog_manager.cc:5495] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f reported cstate change: term changed from 0 to 1, leader changed from <none> to c2f9aa3153fc487c8c5cdb06e521ba8f (127.6.209.195). New cstate: current_term: 1 leader_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } health_report { overall_health: HEALTHY } } }
I20260926 15:15:09.157334 9215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:09.151738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40062 (local address 127.6.209.195:35275)
0926 15:15:09.151943 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:09.151949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:09.152044 (+ 95us) server_negotiation.cc:407] Connection header received
0926 15:15:09.152323 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:09.152328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:09.152412 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:09.152569 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:15:09.152924 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:09.153799 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:09.155611 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:09.155938 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:09.156016 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:09.156442 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:15:09.156883 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:15:09.156967 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:09.157097 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 15:15:09.157147 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 15:15:09.157464 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:09.151727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:35275 (local address 127.0.0.1:40062)
0926 15:15:09.151973 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:09.151992 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:09.152134 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:09.152606 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:09.152611 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:09.152622 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:15:09.152787 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:09.152797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:09.153917 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:09.153923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:09.155468 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:09.155484 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:09.156070 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:09.156075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:09.156232 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:09.156294 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:15:09.156971 (+ 677us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:15:09.156988 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 15:15:09.157076 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 15:15:09.157206 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 15:15:09.163569 9017 raft_consensus.cc:1261] T 656d5a418fc04e19b7b93af652ebb2f6 P 2cd15daf6de446359f72ef4769e57039 [term 1 FOLLOWER]: Refusing update from remote peer c2f9aa3153fc487c8c5cdb06e521ba8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:09.163641 8878 raft_consensus.cc:1261] T 656d5a418fc04e19b7b93af652ebb2f6 P 9caa979a9710444182e0a168d483940e [term 1 FOLLOWER]: Refusing update from remote peer c2f9aa3153fc487c8c5cdb06e521ba8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:09.164424 9244 consensus_queue.cc:1035] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:09.164770 9228 consensus_queue.cc:1035] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:09.181185 9227 tablet_replica.cc:439] T 656d5a418fc04e19b7b93af652ebb2f6 P 9caa979a9710444182e0a168d483940e: TxnStatusTablet state changed. Reason: New leader c2f9aa3153fc487c8c5cdb06e521ba8f. Latest consensus state: current_term: 1 leader_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } } }
I20260926 15:15:09.181459 9227 tablet_replica.cc:442] T 656d5a418fc04e19b7b93af652ebb2f6 P 9caa979a9710444182e0a168d483940e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:15:09.182047 9226 tablet_replica.cc:439] T 656d5a418fc04e19b7b93af652ebb2f6 P 2cd15daf6de446359f72ef4769e57039: TxnStatusTablet state changed. Reason: New leader c2f9aa3153fc487c8c5cdb06e521ba8f. Latest consensus state: current_term: 1 leader_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } } }
I20260926 15:15:09.182312 9226 tablet_replica.cc:442] T 656d5a418fc04e19b7b93af652ebb2f6 P 2cd15daf6de446359f72ef4769e57039: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:15:09.185715 9251 mvcc.cc:205] Tried to move back new op lower bound from 7333624664725839872 to 7333624664591069184. Current Snapshot: MvccSnapshot[applied={T|T < 7333624664725839872}]
I20260926 15:15:09.196154 9228 tablet_replica.cc:439] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } } }
I20260926 15:15:09.196522 9228 tablet_replica.cc:442] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:15:09.197808 9227 tablet_replica.cc:439] T 656d5a418fc04e19b7b93af652ebb2f6 P 9caa979a9710444182e0a168d483940e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } } }
I20260926 15:15:09.198038 9227 tablet_replica.cc:442] T 656d5a418fc04e19b7b93af652ebb2f6 P 9caa979a9710444182e0a168d483940e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:15:09.198181 9250 mvcc.cc:205] Tried to move back new op lower bound from 7333624664725839872 to 7333624664591069184. Current Snapshot: MvccSnapshot[applied={T|T < 7333624664725839872}]
I20260926 15:15:09.201179 9228 tablet_replica.cc:439] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } } }
I20260926 15:15:09.201488 9228 tablet_replica.cc:442] T 656d5a418fc04e19b7b93af652ebb2f6 P c2f9aa3153fc487c8c5cdb06e521ba8f: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:15:09.210893 9226 tablet_replica.cc:439] T 656d5a418fc04e19b7b93af652ebb2f6 P 2cd15daf6de446359f72ef4769e57039: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } } }
I20260926 15:15:09.211159 9226 tablet_replica.cc:442] T 656d5a418fc04e19b7b93af652ebb2f6 P 2cd15daf6de446359f72ef4769e57039: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:15:09.223945 9219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:09.217604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57322 (local address 127.6.209.193:40299)
0926 15:15:09.217806 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:09.217813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:09.218328 (+ 515us) server_negotiation.cc:407] Connection header received
0926 15:15:09.218624 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:09.218629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:09.218700 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:09.218802 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:15:09.219294 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:09.220019 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:09.221985 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:09.222378 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:09.222465 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:09.222921 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:15:09.223371 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:15:09.223472 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:09.223599 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 15:15:09.223656 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 15:15:09.239599 9263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:09.234388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:35275 (local address 127.6.209.193:54139)
0926 15:15:09.234885 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:09.234903 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:09.234999 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:09.235300 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:09.235303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:09.235311 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:15:09.235533 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:09.235538 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:09.236778 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:09.236780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:09.237786 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:09.237795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:09.238692 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:09.238695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:09.238792 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:09.238833 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:15:09.239198 (+ 365us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:15:09.239205 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 15:15:09.239262 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 15:15:09.239341 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":281,"threads_started":1}
I20260926 15:15:09.239666 9215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:09.234334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:54139 (local address 127.6.209.195:35275)
0926 15:15:09.234499 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:09.234503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:09.234980 (+ 477us) server_negotiation.cc:407] Connection header received
0926 15:15:09.235122 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:09.235128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:09.235201 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:09.235288 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:15:09.235653 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:09.236656 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:09.237906 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:09.238556 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:09.238667 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:09.238991 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:15:09.239110 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:15:09.239190 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:09.239304 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 15:15:09.239371 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 15:15:09.248570 9017 raft_consensus.cc:1261] T eb943a8392114ce5ab49b0f8255f5f58 P 2cd15daf6de446359f72ef4769e57039 [term 1 FOLLOWER]: Refusing update from remote peer 9caa979a9710444182e0a168d483940e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:09.248732 9151 raft_consensus.cc:1261] T eb943a8392114ce5ab49b0f8255f5f58 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 1 FOLLOWER]: Refusing update from remote peer 9caa979a9710444182e0a168d483940e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:09.249217 9231 consensus_queue.cc:1035] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:09.249781 9227 consensus_queue.cc:1035] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:09.292305 9219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:09.277691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:58377 (local address 127.6.209.193:40299)
0926 15:15:09.277862 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:09.277868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:09.287384 (+ 9516us) server_negotiation.cc:407] Connection header received
0926 15:15:09.287532 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:09.287535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:09.287583 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:09.287675 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:15:09.288187 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:09.289056 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:09.290868 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:09.291201 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:09.291279 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:09.291705 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:15:09.291796 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:15:09.291856 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:09.292019 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 15:15:09.292083 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 15:15:09.292506 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:09.277350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:40299 (local address 127.6.209.195:58377)
0926 15:15:09.287286 (+ 9936us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:09.287303 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:09.287427 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:09.287686 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:09.287691 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:09.287701 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:15:09.288032 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:09.288040 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:09.289190 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:09.289194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:09.290755 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:09.290768 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:09.291358 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:09.291364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:09.291520 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:09.291574 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:15:09.291880 (+ 306us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:15:09.291889 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 15:15:09.291973 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 15:15:09.292101 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9762,"thread_start_us":10171,"threads_started":1}
I20260926 15:15:09.324307 8717 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:09.324635 8717 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:09.328347 8717 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1: Sending DeleteTablet for 3 replicas of tablet eb943a8392114ce5ab49b0f8255f5f58
I20260926 15:15:09.329331 8857 tablet_service.cc:1514] Processing DeleteTablet for tablet eb943a8392114ce5ab49b0f8255f5f58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:09 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:57308
I20260926 15:15:09.329464 8992 tablet_service.cc:1514] Processing DeleteTablet for tablet eb943a8392114ce5ab49b0f8255f5f58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:09 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:36248
I20260926 15:15:09.330209 9131 tablet_service.cc:1514] Processing DeleteTablet for tablet eb943a8392114ce5ab49b0f8255f5f58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:09 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:40046
I20260926 15:15:09.336016 8717 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:15:09.337637 8717 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:15:09.347230 8991 tablet_service.cc:1467] Processing CreateTablet for tablet 9c69c20d6f794741991a545e3e5d688f (DEFAULT_TABLE table=test-table [id=24ae483f657442cf9b2bf62391744d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:09.347822 8991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c69c20d6f794741991a545e3e5d688f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:09.347755 9130 tablet_service.cc:1467] Processing CreateTablet for tablet 9c69c20d6f794741991a545e3e5d688f (DEFAULT_TABLE table=test-table [id=24ae483f657442cf9b2bf62391744d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:09.348258 9130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c69c20d6f794741991a545e3e5d688f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:09.351336 9273 tablet_replica.cc:330] T eb943a8392114ce5ab49b0f8255f5f58 P 2cd15daf6de446359f72ef4769e57039: stopping tablet replica
I20260926 15:15:09.351775 9273 raft_consensus.cc:2229] T eb943a8392114ce5ab49b0f8255f5f58 P 2cd15daf6de446359f72ef4769e57039 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:09.352274 9273 raft_consensus.cc:2258] T eb943a8392114ce5ab49b0f8255f5f58 P 2cd15daf6de446359f72ef4769e57039 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:09.353214 8856 tablet_service.cc:1467] Processing CreateTablet for tablet 9c69c20d6f794741991a545e3e5d688f (DEFAULT_TABLE table=test-table [id=24ae483f657442cf9b2bf62391744d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:09.353734 8856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c69c20d6f794741991a545e3e5d688f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:09.354058 9273 ts_tablet_manager.cc:1905] T eb943a8392114ce5ab49b0f8255f5f58 P 2cd15daf6de446359f72ef4769e57039: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:09.355068 9224 tablet_bootstrap.cc:492] T 9c69c20d6f794741991a545e3e5d688f P c2f9aa3153fc487c8c5cdb06e521ba8f: Bootstrap starting.
I20260926 15:15:09.356602 9224 tablet_bootstrap.cc:654] T 9c69c20d6f794741991a545e3e5d688f P c2f9aa3153fc487c8c5cdb06e521ba8f: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:09.360257 9273 ts_tablet_manager.cc:1918] T eb943a8392114ce5ab49b0f8255f5f58 P 2cd15daf6de446359f72ef4769e57039: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:15:09.360387 9273 log.cc:1196] T eb943a8392114ce5ab49b0f8255f5f58 P 2cd15daf6de446359f72ef4769e57039: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/eb943a8392114ce5ab49b0f8255f5f58
I20260926 15:15:09.360845 9273 ts_tablet_manager.cc:1939] T eb943a8392114ce5ab49b0f8255f5f58 P 2cd15daf6de446359f72ef4769e57039: Deleting consensus metadata
I20260926 15:15:09.361024 9221 tablet_bootstrap.cc:492] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e: Bootstrap starting.
I20260926 15:15:09.362702 9221 tablet_bootstrap.cc:654] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:09.362738 8701 catalog_manager.cc:4841] TS 2cd15daf6de446359f72ef4769e57039 (127.6.209.194:33079): tablet eb943a8392114ce5ab49b0f8255f5f58 (table test-table [id=931c602e898b4e41ad3f10cc7e005d1c]) successfully deleted
I20260926 15:15:09.363577 9274 tablet_replica.cc:330] T eb943a8392114ce5ab49b0f8255f5f58 P c2f9aa3153fc487c8c5cdb06e521ba8f: stopping tablet replica
I20260926 15:15:09.364123 9272 tablet_replica.cc:330] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e: stopping tablet replica
I20260926 15:15:09.364501 9272 raft_consensus.cc:2229] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:15:09.365051 9272 raft_consensus.cc:2258] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:09.365510 9220 tablet_bootstrap.cc:492] T 9c69c20d6f794741991a545e3e5d688f P 2cd15daf6de446359f72ef4769e57039: Bootstrap starting.
I20260926 15:15:09.365608 9274 raft_consensus.cc:2229] T eb943a8392114ce5ab49b0f8255f5f58 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:09.365864 9274 raft_consensus.cc:2258] T eb943a8392114ce5ab49b0f8255f5f58 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:09.366938 9220 tablet_bootstrap.cc:654] T 9c69c20d6f794741991a545e3e5d688f P 2cd15daf6de446359f72ef4769e57039: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:09.367424 9274 ts_tablet_manager.cc:1905] T eb943a8392114ce5ab49b0f8255f5f58 P c2f9aa3153fc487c8c5cdb06e521ba8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:09.370278 9272 ts_tablet_manager.cc:1905] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:09.371402 9221 tablet_bootstrap.cc:492] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e: No bootstrap required, opened a new log
I20260926 15:15:09.371598 9221 ts_tablet_manager.cc:1397] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.002s
I20260926 15:15:09.372331 9221 raft_consensus.cc:348] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:09.372555 9221 raft_consensus.cc:374] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:09.372627 9221 raft_consensus.cc:729] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9caa979a9710444182e0a168d483940e, State: Initialized, Role: FOLLOWER
I20260926 15:15:09.372925 9221 consensus_queue.cc:260] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:09.376201 9221 ts_tablet_manager.cc:1428] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 15:15:09.378868 9220 tablet_bootstrap.cc:492] T 9c69c20d6f794741991a545e3e5d688f P 2cd15daf6de446359f72ef4769e57039: No bootstrap required, opened a new log
I20260926 15:15:09.379019 9220 ts_tablet_manager.cc:1397] T 9c69c20d6f794741991a545e3e5d688f P 2cd15daf6de446359f72ef4769e57039: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.003s
I20260926 15:15:09.379695 9220 raft_consensus.cc:348] T 9c69c20d6f794741991a545e3e5d688f P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:09.379906 9220 raft_consensus.cc:374] T 9c69c20d6f794741991a545e3e5d688f P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:09.379985 9220 raft_consensus.cc:729] T 9c69c20d6f794741991a545e3e5d688f P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd15daf6de446359f72ef4769e57039, State: Initialized, Role: FOLLOWER
I20260926 15:15:09.380009 9274 ts_tablet_manager.cc:1918] T eb943a8392114ce5ab49b0f8255f5f58 P c2f9aa3153fc487c8c5cdb06e521ba8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:15:09.380152 9274 log.cc:1196] T eb943a8392114ce5ab49b0f8255f5f58 P c2f9aa3153fc487c8c5cdb06e521ba8f: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/eb943a8392114ce5ab49b0f8255f5f58
I20260926 15:15:09.380218 9220 consensus_queue.cc:260] T 9c69c20d6f794741991a545e3e5d688f P 2cd15daf6de446359f72ef4769e57039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:09.380620 9274 ts_tablet_manager.cc:1939] T eb943a8392114ce5ab49b0f8255f5f58 P c2f9aa3153fc487c8c5cdb06e521ba8f: Deleting consensus metadata
I20260926 15:15:09.382222 8704 catalog_manager.cc:4841] TS c2f9aa3153fc487c8c5cdb06e521ba8f (127.6.209.195:35275): tablet eb943a8392114ce5ab49b0f8255f5f58 (table test-table [id=931c602e898b4e41ad3f10cc7e005d1c]) successfully deleted
I20260926 15:15:09.382490 9224 tablet_bootstrap.cc:492] T 9c69c20d6f794741991a545e3e5d688f P c2f9aa3153fc487c8c5cdb06e521ba8f: No bootstrap required, opened a new log
I20260926 15:15:09.382642 9224 ts_tablet_manager.cc:1397] T 9c69c20d6f794741991a545e3e5d688f P c2f9aa3153fc487c8c5cdb06e521ba8f: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.000s
I20260926 15:15:09.382968 9220 ts_tablet_manager.cc:1428] T 9c69c20d6f794741991a545e3e5d688f P 2cd15daf6de446359f72ef4769e57039: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260926 15:15:09.383319 9224 raft_consensus.cc:348] T 9c69c20d6f794741991a545e3e5d688f P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:09.383577 9224 raft_consensus.cc:374] T 9c69c20d6f794741991a545e3e5d688f P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:09.383664 9224 raft_consensus.cc:729] T 9c69c20d6f794741991a545e3e5d688f P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2f9aa3153fc487c8c5cdb06e521ba8f, State: Initialized, Role: FOLLOWER
I20260926 15:15:09.383910 9224 consensus_queue.cc:260] T 9c69c20d6f794741991a545e3e5d688f P c2f9aa3153fc487c8c5cdb06e521ba8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:09.385308 9224 ts_tablet_manager.cc:1428] T 9c69c20d6f794741991a545e3e5d688f P c2f9aa3153fc487c8c5cdb06e521ba8f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 15:15:09.388599 9272 ts_tablet_manager.cc:1918] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:15:09.388743 9272 log.cc:1196] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-0/wal/wals/eb943a8392114ce5ab49b0f8255f5f58
I20260926 15:15:09.389214 9272 ts_tablet_manager.cc:1939] T eb943a8392114ce5ab49b0f8255f5f58 P 9caa979a9710444182e0a168d483940e: Deleting consensus metadata
I20260926 15:15:09.390479 8704 catalog_manager.cc:4841] TS 9caa979a9710444182e0a168d483940e (127.6.209.193:40299): tablet eb943a8392114ce5ab49b0f8255f5f58 (table test-table [id=931c602e898b4e41ad3f10cc7e005d1c]) successfully deleted
I20260926 15:15:09.394455 9231 raft_consensus.cc:482] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:15:09.394646 9231 raft_consensus.cc:504] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:09.395200 9231 leader_election.cc:288] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cd15daf6de446359f72ef4769e57039 (127.6.209.194:33079), c2f9aa3153fc487c8c5cdb06e521ba8f (127.6.209.195:35275)
I20260926 15:15:09.395684 9017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c69c20d6f794741991a545e3e5d688f" candidate_uuid: "9caa979a9710444182e0a168d483940e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd15daf6de446359f72ef4769e57039" is_pre_election: true
I20260926 15:15:09.395751 9151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c69c20d6f794741991a545e3e5d688f" candidate_uuid: "9caa979a9710444182e0a168d483940e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" is_pre_election: true
I20260926 15:15:09.395865 9017 raft_consensus.cc:2454] T 9c69c20d6f794741991a545e3e5d688f P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9caa979a9710444182e0a168d483940e in term 0.
I20260926 15:15:09.395938 9151 raft_consensus.cc:2454] T 9c69c20d6f794741991a545e3e5d688f P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9caa979a9710444182e0a168d483940e in term 0.
I20260926 15:15:09.396214 8804 leader_election.cc:302] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cd15daf6de446359f72ef4769e57039, 9caa979a9710444182e0a168d483940e; no voters:
I20260926 15:15:09.396508 9231 raft_consensus.cc:2781] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:15:09.396600 9231 raft_consensus.cc:482] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:15:09.396688 9231 raft_consensus.cc:3037] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:09.399189 9231 raft_consensus.cc:504] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:09.399962 9231 leader_election.cc:288] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e [CANDIDATE]: Term 1 election: Requested vote from peers 2cd15daf6de446359f72ef4769e57039 (127.6.209.194:33079), c2f9aa3153fc487c8c5cdb06e521ba8f (127.6.209.195:35275)
I20260926 15:15:09.400030 9017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c69c20d6f794741991a545e3e5d688f" candidate_uuid: "9caa979a9710444182e0a168d483940e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd15daf6de446359f72ef4769e57039"
I20260926 15:15:09.400177 9017 raft_consensus.cc:3037] T 9c69c20d6f794741991a545e3e5d688f P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:09.400266 9151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c69c20d6f794741991a545e3e5d688f" candidate_uuid: "9caa979a9710444182e0a168d483940e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f"
I20260926 15:15:09.400432 9151 raft_consensus.cc:3037] T 9c69c20d6f794741991a545e3e5d688f P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:09.402849 9017 raft_consensus.cc:2454] T 9c69c20d6f794741991a545e3e5d688f P 2cd15daf6de446359f72ef4769e57039 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9caa979a9710444182e0a168d483940e in term 1.
I20260926 15:15:09.403230 8804 leader_election.cc:302] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cd15daf6de446359f72ef4769e57039, 9caa979a9710444182e0a168d483940e; no voters:
I20260926 15:15:09.403486 9231 raft_consensus.cc:2781] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:09.403803 9151 raft_consensus.cc:2454] T 9c69c20d6f794741991a545e3e5d688f P c2f9aa3153fc487c8c5cdb06e521ba8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9caa979a9710444182e0a168d483940e in term 1.
I20260926 15:15:09.403946 9231 raft_consensus.cc:686] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e [term 1 LEADER]: Becoming Leader. State: Replica: 9caa979a9710444182e0a168d483940e, State: Running, Role: LEADER
I20260926 15:15:09.404206 9231 consensus_queue.cc:237] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:09.406857 8717 catalog_manager.cc:5495] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9caa979a9710444182e0a168d483940e (127.6.209.193). New cstate: current_term: 1 leader_uuid: "9caa979a9710444182e0a168d483940e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } health_report { overall_health: UNKNOWN } } }
I20260926 15:15:09.436671 9151 raft_consensus.cc:1261] T 9c69c20d6f794741991a545e3e5d688f P c2f9aa3153fc487c8c5cdb06e521ba8f [term 1 FOLLOWER]: Refusing update from remote peer 9caa979a9710444182e0a168d483940e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:09.437322 9016 raft_consensus.cc:1261] T 9c69c20d6f794741991a545e3e5d688f P 2cd15daf6de446359f72ef4769e57039 [term 1 FOLLOWER]: Refusing update from remote peer 9caa979a9710444182e0a168d483940e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:09.437238 9231 consensus_queue.cc:1035] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:09.437870 9269 consensus_queue.cc:1035] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 15:15:09.471517 9251 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 15:15:11.071676 8717 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:11.071939 8717 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:11.075255 8717 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1: Sending DeleteTablet for 3 replicas of tablet 9c69c20d6f794741991a545e3e5d688f
I20260926 15:15:11.075804 8857 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c69c20d6f794741991a545e3e5d688f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:11 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:57308
I20260926 15:15:11.075935 8991 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c69c20d6f794741991a545e3e5d688f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:11 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:36248
I20260926 15:15:11.075997 9130 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c69c20d6f794741991a545e3e5d688f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:11 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:40046
I20260926 15:15:11.085958 9305 tablet_replica.cc:330] T 9c69c20d6f794741991a545e3e5d688f P c2f9aa3153fc487c8c5cdb06e521ba8f: stopping tablet replica
I20260926 15:15:11.086485 9305 raft_consensus.cc:2229] T 9c69c20d6f794741991a545e3e5d688f P c2f9aa3153fc487c8c5cdb06e521ba8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:11.086565 9303 tablet_replica.cc:330] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e: stopping tablet replica
I20260926 15:15:11.086791 9303 raft_consensus.cc:2229] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:15:11.086936 9305 raft_consensus.cc:2258] T 9c69c20d6f794741991a545e3e5d688f P c2f9aa3153fc487c8c5cdb06e521ba8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:11.087199 9303 raft_consensus.cc:2258] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:11.088132 9303 ts_tablet_manager.cc:1905] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:11.092342 9304 tablet_replica.cc:330] T 9c69c20d6f794741991a545e3e5d688f P 2cd15daf6de446359f72ef4769e57039: stopping tablet replica
I20260926 15:15:11.092576 9304 raft_consensus.cc:2229] T 9c69c20d6f794741991a545e3e5d688f P 2cd15daf6de446359f72ef4769e57039 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:11.092856 9304 raft_consensus.cc:2258] T 9c69c20d6f794741991a545e3e5d688f P 2cd15daf6de446359f72ef4769e57039 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:11.093667 9305 ts_tablet_manager.cc:1905] T 9c69c20d6f794741991a545e3e5d688f P c2f9aa3153fc487c8c5cdb06e521ba8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:11.094991 9303 ts_tablet_manager.cc:1918] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 15:15:11.095147 9303 log.cc:1196] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-0/wal/wals/9c69c20d6f794741991a545e3e5d688f
I20260926 15:15:11.095539 9303 ts_tablet_manager.cc:1939] T 9c69c20d6f794741991a545e3e5d688f P 9caa979a9710444182e0a168d483940e: Deleting consensus metadata
I20260926 15:15:11.096411 8704 catalog_manager.cc:4841] TS 9caa979a9710444182e0a168d483940e (127.6.209.193:40299): tablet 9c69c20d6f794741991a545e3e5d688f (table test-table [id=24ae483f657442cf9b2bf62391744d9d]) successfully deleted
I20260926 15:15:11.099975 9305 ts_tablet_manager.cc:1918] T 9c69c20d6f794741991a545e3e5d688f P c2f9aa3153fc487c8c5cdb06e521ba8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 15:15:11.100250 9305 log.cc:1196] T 9c69c20d6f794741991a545e3e5d688f P c2f9aa3153fc487c8c5cdb06e521ba8f: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/9c69c20d6f794741991a545e3e5d688f
I20260926 15:15:11.100800 9304 ts_tablet_manager.cc:1905] T 9c69c20d6f794741991a545e3e5d688f P 2cd15daf6de446359f72ef4769e57039: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:11.107194 9304 ts_tablet_manager.cc:1918] T 9c69c20d6f794741991a545e3e5d688f P 2cd15daf6de446359f72ef4769e57039: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 15:15:11.107635 9305 ts_tablet_manager.cc:1939] T 9c69c20d6f794741991a545e3e5d688f P c2f9aa3153fc487c8c5cdb06e521ba8f: Deleting consensus metadata
I20260926 15:15:11.108420 9304 log.cc:1196] T 9c69c20d6f794741991a545e3e5d688f P 2cd15daf6de446359f72ef4769e57039: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/9c69c20d6f794741991a545e3e5d688f
I20260926 15:15:11.108539 8704 catalog_manager.cc:4841] TS c2f9aa3153fc487c8c5cdb06e521ba8f (127.6.209.195:35275): tablet 9c69c20d6f794741991a545e3e5d688f (table test-table [id=24ae483f657442cf9b2bf62391744d9d]) successfully deleted
I20260926 15:15:11.108862 9304 ts_tablet_manager.cc:1939] T 9c69c20d6f794741991a545e3e5d688f P 2cd15daf6de446359f72ef4769e57039: Deleting consensus metadata
I20260926 15:15:11.109727 8701 catalog_manager.cc:4841] TS 2cd15daf6de446359f72ef4769e57039 (127.6.209.194:33079): tablet 9c69c20d6f794741991a545e3e5d688f (table test-table [id=24ae483f657442cf9b2bf62391744d9d]) successfully deleted
Sep 26 15:15:11 dist-test-slave-j6q7 krb5kdc[8672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 15:15:11 dist-test-slave-j6q7 krb5kdc[8672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:11.145982 9313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:11.126013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33484 (local address 127.6.209.254:37989)
0926 15:15:11.126542 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:11.126547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:11.126666 (+ 119us) server_negotiation.cc:407] Connection header received
0926 15:15:11.126859 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:11.126862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:11.126915 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:11.127018 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:11.128234 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.128986 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.130088 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.130414 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.130528 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:11.141313 (+ 10785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:11.141318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:11.141321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:11.141350 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:11.143953 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:11.144480 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:11.144483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:11.144485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:11.144589 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:11.144922 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:11.144925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:11.144927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:11.145153 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:11.145297 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:11.145511 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 15:15:11.145659 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":286,"threads_started":1}
I20260926 15:15:11.148686 8717 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:15:11.149253 8717 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:15:11.155049 8857 tablet_service.cc:1467] Processing CreateTablet for tablet b36aa379d04a4f4db93270d302001054 (DEFAULT_TABLE table=test-table [id=01b093943376462d801873d02838ef60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:11.155337 9130 tablet_service.cc:1467] Processing CreateTablet for tablet b36aa379d04a4f4db93270d302001054 (DEFAULT_TABLE table=test-table [id=01b093943376462d801873d02838ef60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:11.155457 8857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b36aa379d04a4f4db93270d302001054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:11.155741 9130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b36aa379d04a4f4db93270d302001054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:11.160720 9314 tablet_bootstrap.cc:492] T b36aa379d04a4f4db93270d302001054 P 9caa979a9710444182e0a168d483940e: Bootstrap starting.
I20260926 15:15:11.160728 8991 tablet_service.cc:1467] Processing CreateTablet for tablet b36aa379d04a4f4db93270d302001054 (DEFAULT_TABLE table=test-table [id=01b093943376462d801873d02838ef60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:11.161199 8991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b36aa379d04a4f4db93270d302001054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:11.162895 9314 tablet_bootstrap.cc:654] T b36aa379d04a4f4db93270d302001054 P 9caa979a9710444182e0a168d483940e: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:11.165012 9315 tablet_bootstrap.cc:492] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f: Bootstrap starting.
I20260926 15:15:11.165136 9314 tablet_bootstrap.cc:492] T b36aa379d04a4f4db93270d302001054 P 9caa979a9710444182e0a168d483940e: No bootstrap required, opened a new log
I20260926 15:15:11.165637 9314 ts_tablet_manager.cc:1397] T b36aa379d04a4f4db93270d302001054 P 9caa979a9710444182e0a168d483940e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 15:15:11.166339 9317 tablet_bootstrap.cc:492] T b36aa379d04a4f4db93270d302001054 P 2cd15daf6de446359f72ef4769e57039: Bootstrap starting.
I20260926 15:15:11.166433 9314 raft_consensus.cc:348] T b36aa379d04a4f4db93270d302001054 P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:11.166713 9314 raft_consensus.cc:374] T b36aa379d04a4f4db93270d302001054 P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:11.166836 9314 raft_consensus.cc:729] T b36aa379d04a4f4db93270d302001054 P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9caa979a9710444182e0a168d483940e, State: Initialized, Role: FOLLOWER
I20260926 15:15:11.166867 9315 tablet_bootstrap.cc:654] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:11.167080 9314 consensus_queue.cc:260] T b36aa379d04a4f4db93270d302001054 P 9caa979a9710444182e0a168d483940e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:11.167508 9314 ts_tablet_manager.cc:1428] T b36aa379d04a4f4db93270d302001054 P 9caa979a9710444182e0a168d483940e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:15:11.168011 9317 tablet_bootstrap.cc:654] T b36aa379d04a4f4db93270d302001054 P 2cd15daf6de446359f72ef4769e57039: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:11.168684 9315 tablet_bootstrap.cc:492] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f: No bootstrap required, opened a new log
I20260926 15:15:11.168810 9315 ts_tablet_manager.cc:1397] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260926 15:15:11.169526 9315 raft_consensus.cc:348] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:11.169739 9315 raft_consensus.cc:374] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:11.169816 9315 raft_consensus.cc:729] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2f9aa3153fc487c8c5cdb06e521ba8f, State: Initialized, Role: FOLLOWER
I20260926 15:15:11.170042 9315 consensus_queue.cc:260] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:11.170220 9317 tablet_bootstrap.cc:492] T b36aa379d04a4f4db93270d302001054 P 2cd15daf6de446359f72ef4769e57039: No bootstrap required, opened a new log
I20260926 15:15:11.170359 9317 ts_tablet_manager.cc:1397] T b36aa379d04a4f4db93270d302001054 P 2cd15daf6de446359f72ef4769e57039: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 15:15:11.170531 9315 ts_tablet_manager.cc:1428] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260926 15:15:11.171475 9317 raft_consensus.cc:348] T b36aa379d04a4f4db93270d302001054 P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:11.171676 9317 raft_consensus.cc:374] T b36aa379d04a4f4db93270d302001054 P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:11.171800 9317 raft_consensus.cc:729] T b36aa379d04a4f4db93270d302001054 P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd15daf6de446359f72ef4769e57039, State: Initialized, Role: FOLLOWER
I20260926 15:15:11.172003 9317 consensus_queue.cc:260] T b36aa379d04a4f4db93270d302001054 P 2cd15daf6de446359f72ef4769e57039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:11.172713 9317 ts_tablet_manager.cc:1428] T b36aa379d04a4f4db93270d302001054 P 2cd15daf6de446359f72ef4769e57039: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:15:11.232415 9283 raft_consensus.cc:482] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:15:11.232573 9283 raft_consensus.cc:504] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:11.233186 9283 leader_election.cc:288] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9caa979a9710444182e0a168d483940e (127.6.209.193:40299), 2cd15daf6de446359f72ef4769e57039 (127.6.209.194:33079)
I20260926 15:15:11.233675 9016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b36aa379d04a4f4db93270d302001054" candidate_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd15daf6de446359f72ef4769e57039" is_pre_election: true
I20260926 15:15:11.233695 8878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b36aa379d04a4f4db93270d302001054" candidate_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9caa979a9710444182e0a168d483940e" is_pre_election: true
I20260926 15:15:11.233906 8878 raft_consensus.cc:2454] T b36aa379d04a4f4db93270d302001054 P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2f9aa3153fc487c8c5cdb06e521ba8f in term 0.
I20260926 15:15:11.233924 9016 raft_consensus.cc:2454] T b36aa379d04a4f4db93270d302001054 P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2f9aa3153fc487c8c5cdb06e521ba8f in term 0.
I20260926 15:15:11.234369 9085 leader_election.cc:302] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2cd15daf6de446359f72ef4769e57039, 9caa979a9710444182e0a168d483940e, c2f9aa3153fc487c8c5cdb06e521ba8f; no voters:
I20260926 15:15:11.234668 9283 raft_consensus.cc:2781] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:15:11.234791 9283 raft_consensus.cc:482] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:15:11.234895 9283 raft_consensus.cc:3037] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:11.236667 9283 raft_consensus.cc:504] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:11.237211 9283 leader_election.cc:288] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f [CANDIDATE]: Term 1 election: Requested vote from peers 9caa979a9710444182e0a168d483940e (127.6.209.193:40299), 2cd15daf6de446359f72ef4769e57039 (127.6.209.194:33079)
I20260926 15:15:11.237497 8878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b36aa379d04a4f4db93270d302001054" candidate_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9caa979a9710444182e0a168d483940e"
I20260926 15:15:11.237614 9016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b36aa379d04a4f4db93270d302001054" candidate_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd15daf6de446359f72ef4769e57039"
I20260926 15:15:11.237680 8878 raft_consensus.cc:3037] T b36aa379d04a4f4db93270d302001054 P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:11.237818 9016 raft_consensus.cc:3037] T b36aa379d04a4f4db93270d302001054 P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:11.239428 8878 raft_consensus.cc:2454] T b36aa379d04a4f4db93270d302001054 P 9caa979a9710444182e0a168d483940e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2f9aa3153fc487c8c5cdb06e521ba8f in term 1.
I20260926 15:15:11.239594 9016 raft_consensus.cc:2454] T b36aa379d04a4f4db93270d302001054 P 2cd15daf6de446359f72ef4769e57039 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2f9aa3153fc487c8c5cdb06e521ba8f in term 1.
I20260926 15:15:11.239753 9085 leader_election.cc:302] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9caa979a9710444182e0a168d483940e, c2f9aa3153fc487c8c5cdb06e521ba8f; no voters:
I20260926 15:15:11.240029 9283 raft_consensus.cc:2781] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:11.240160 9283 raft_consensus.cc:686] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 1 LEADER]: Becoming Leader. State: Replica: c2f9aa3153fc487c8c5cdb06e521ba8f, State: Running, Role: LEADER
I20260926 15:15:11.240397 9283 consensus_queue.cc:237] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:11.242780 8717 catalog_manager.cc:5495] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f reported cstate change: term changed from 0 to 1, leader changed from <none> to c2f9aa3153fc487c8c5cdb06e521ba8f (127.6.209.195). New cstate: current_term: 1 leader_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } health_report { overall_health: HEALTHY } } }
I20260926 15:15:11.271646 9323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:11.267698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40072 (local address 127.6.209.195:35275)
0926 15:15:11.268108 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:11.268112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:11.268128 (+ 16us) server_negotiation.cc:407] Connection header received
0926 15:15:11.268169 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:11.268172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:11.268217 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:11.268305 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:15:11.268561 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.269271 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.270374 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.270681 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.270758 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:11.271064 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:15:11.271144 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:15:11.271199 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:11.271313 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 15:15:11.271359 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":225,"threads_started":1}
I20260926 15:15:11.281688 9323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:11.277909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:44231 (local address 127.6.209.195:35275)
0926 15:15:11.278060 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:11.278068 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:11.278328 (+ 260us) server_negotiation.cc:407] Connection header received
0926 15:15:11.278423 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:11.278426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:11.278470 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:11.278534 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:15:11.278828 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.279399 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.280510 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.280835 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.280911 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:11.281235 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:15:11.281306 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:15:11.281365 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:11.281472 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 15:15:11.281525 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 15:15:11.281824 9326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:11.277871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:35275 (local address 127.6.209.195:44231)
0926 15:15:11.278256 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:11.278268 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:11.278351 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:11.278554 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:11.278557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:11.278564 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:15:11.278725 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:11.278730 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.279486 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.279489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:11.280395 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:11.280416 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.280948 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.280952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:11.281048 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:11.281075 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:15:11.281382 (+ 307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:15:11.281389 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 15:15:11.281443 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 15:15:11.281531 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":208,"threads_started":1}
I20260926 15:15:11.287950 9016 raft_consensus.cc:1261] T b36aa379d04a4f4db93270d302001054 P 2cd15daf6de446359f72ef4769e57039 [term 1 FOLLOWER]: Refusing update from remote peer c2f9aa3153fc487c8c5cdb06e521ba8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:11.287976 8878 raft_consensus.cc:1261] T b36aa379d04a4f4db93270d302001054 P 9caa979a9710444182e0a168d483940e [term 1 FOLLOWER]: Refusing update from remote peer c2f9aa3153fc487c8c5cdb06e521ba8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:11.288373 9283 consensus_queue.cc:1035] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:11.288626 9298 consensus_queue.cc:1035] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:11.291289 9281 mvcc.cc:205] Tried to move back new op lower bound from 7333624673431638016 to 7333624673242648576. Current Snapshot: MvccSnapshot[applied={T|T < 7333624673431638016}]
I20260926 15:15:11.329704 8715 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:11.329914 8715 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:11.332224 8715 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1: Sending DeleteTablet for 3 replicas of tablet b36aa379d04a4f4db93270d302001054
I20260926 15:15:11.332850 8991 tablet_service.cc:1514] Processing DeleteTablet for tablet b36aa379d04a4f4db93270d302001054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:11 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:36248
I20260926 15:15:11.332830 8857 tablet_service.cc:1514] Processing DeleteTablet for tablet b36aa379d04a4f4db93270d302001054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:11 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:57308
I20260926 15:15:11.333065 9304 tablet_replica.cc:330] T b36aa379d04a4f4db93270d302001054 P 2cd15daf6de446359f72ef4769e57039: stopping tablet replica
I20260926 15:15:11.333222 9304 raft_consensus.cc:2229] T b36aa379d04a4f4db93270d302001054 P 2cd15daf6de446359f72ef4769e57039 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:11.333824 9304 raft_consensus.cc:2258] T b36aa379d04a4f4db93270d302001054 P 2cd15daf6de446359f72ef4769e57039 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:11.333869 8715 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 15:15:11.334102 9303 tablet_replica.cc:330] T b36aa379d04a4f4db93270d302001054 P 9caa979a9710444182e0a168d483940e: stopping tablet replica
I20260926 15:15:11.334102 9130 tablet_service.cc:1514] Processing DeleteTablet for tablet b36aa379d04a4f4db93270d302001054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:11 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:40046
I20260926 15:15:11.334283 9303 raft_consensus.cc:2229] T b36aa379d04a4f4db93270d302001054 P 9caa979a9710444182e0a168d483940e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:11.334406 9303 raft_consensus.cc:2258] T b36aa379d04a4f4db93270d302001054 P 9caa979a9710444182e0a168d483940e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 15:15:11.334421 8715 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:15:11.334797 9305 tablet_replica.cc:330] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f: stopping tablet replica
I20260926 15:15:11.334915 9305 raft_consensus.cc:2229] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:15:11.335181 9305 raft_consensus.cc:2258] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:11.336123 9304 ts_tablet_manager.cc:1905] T b36aa379d04a4f4db93270d302001054 P 2cd15daf6de446359f72ef4769e57039: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:11.336130 9305 ts_tablet_manager.cc:1905] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:11.340312 8991 tablet_service.cc:1467] Processing CreateTablet for tablet 9b3328532c244f4ab687bb3ef1d23b5c (DEFAULT_TABLE table=test-table [id=cf214486f57744228c777f2b82f83b08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:11.340412 8857 tablet_service.cc:1467] Processing CreateTablet for tablet 9b3328532c244f4ab687bb3ef1d23b5c (DEFAULT_TABLE table=test-table [id=cf214486f57744228c777f2b82f83b08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:11.340636 8991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b3328532c244f4ab687bb3ef1d23b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:11.340759 8857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b3328532c244f4ab687bb3ef1d23b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:11.341455 9303 ts_tablet_manager.cc:1905] T b36aa379d04a4f4db93270d302001054 P 9caa979a9710444182e0a168d483940e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:11.346233 9130 tablet_service.cc:1467] Processing CreateTablet for tablet 9b3328532c244f4ab687bb3ef1d23b5c (DEFAULT_TABLE table=test-table [id=cf214486f57744228c777f2b82f83b08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:11.346732 9130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b3328532c244f4ab687bb3ef1d23b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:11.349103 9305 ts_tablet_manager.cc:1918] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:15:11.349221 9305 log.cc:1196] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/b36aa379d04a4f4db93270d302001054
I20260926 15:15:11.349638 9305 ts_tablet_manager.cc:1939] T b36aa379d04a4f4db93270d302001054 P c2f9aa3153fc487c8c5cdb06e521ba8f: Deleting consensus metadata
I20260926 15:15:11.350534 8704 catalog_manager.cc:4841] TS c2f9aa3153fc487c8c5cdb06e521ba8f (127.6.209.195:35275): tablet b36aa379d04a4f4db93270d302001054 (table test-table [id=01b093943376462d801873d02838ef60]) successfully deleted
I20260926 15:15:11.350853 9314 tablet_bootstrap.cc:492] T 9b3328532c244f4ab687bb3ef1d23b5c P 9caa979a9710444182e0a168d483940e: Bootstrap starting.
I20260926 15:15:11.352195 9314 tablet_bootstrap.cc:654] T 9b3328532c244f4ab687bb3ef1d23b5c P 9caa979a9710444182e0a168d483940e: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:11.355178 9317 tablet_bootstrap.cc:492] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039: Bootstrap starting.
I20260926 15:15:11.355527 9315 tablet_bootstrap.cc:492] T 9b3328532c244f4ab687bb3ef1d23b5c P c2f9aa3153fc487c8c5cdb06e521ba8f: Bootstrap starting.
I20260926 15:15:11.356731 9317 tablet_bootstrap.cc:654] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:11.357533 9315 tablet_bootstrap.cc:654] T 9b3328532c244f4ab687bb3ef1d23b5c P c2f9aa3153fc487c8c5cdb06e521ba8f: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:11.360268 9303 ts_tablet_manager.cc:1918] T b36aa379d04a4f4db93270d302001054 P 9caa979a9710444182e0a168d483940e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:15:11.360571 9303 log.cc:1196] T b36aa379d04a4f4db93270d302001054 P 9caa979a9710444182e0a168d483940e: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-0/wal/wals/b36aa379d04a4f4db93270d302001054
I20260926 15:15:11.360713 9314 tablet_bootstrap.cc:492] T 9b3328532c244f4ab687bb3ef1d23b5c P 9caa979a9710444182e0a168d483940e: No bootstrap required, opened a new log
I20260926 15:15:11.360981 9314 ts_tablet_manager.cc:1397] T 9b3328532c244f4ab687bb3ef1d23b5c P 9caa979a9710444182e0a168d483940e: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20260926 15:15:11.361570 9303 ts_tablet_manager.cc:1939] T b36aa379d04a4f4db93270d302001054 P 9caa979a9710444182e0a168d483940e: Deleting consensus metadata
I20260926 15:15:11.361979 9314 raft_consensus.cc:348] T 9b3328532c244f4ab687bb3ef1d23b5c P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:11.362284 9314 raft_consensus.cc:374] T 9b3328532c244f4ab687bb3ef1d23b5c P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:11.362728 9304 ts_tablet_manager.cc:1918] T b36aa379d04a4f4db93270d302001054 P 2cd15daf6de446359f72ef4769e57039: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:15:11.362864 9304 log.cc:1196] T b36aa379d04a4f4db93270d302001054 P 2cd15daf6de446359f72ef4769e57039: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/b36aa379d04a4f4db93270d302001054
I20260926 15:15:11.362884 9314 raft_consensus.cc:729] T 9b3328532c244f4ab687bb3ef1d23b5c P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9caa979a9710444182e0a168d483940e, State: Initialized, Role: FOLLOWER
I20260926 15:15:11.363204 9314 consensus_queue.cc:260] T 9b3328532c244f4ab687bb3ef1d23b5c P 9caa979a9710444182e0a168d483940e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:11.363601 9304 ts_tablet_manager.cc:1939] T b36aa379d04a4f4db93270d302001054 P 2cd15daf6de446359f72ef4769e57039: Deleting consensus metadata
I20260926 15:15:11.363870 9314 ts_tablet_manager.cc:1428] T 9b3328532c244f4ab687bb3ef1d23b5c P 9caa979a9710444182e0a168d483940e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260926 15:15:11.364682 8704 catalog_manager.cc:4841] TS 9caa979a9710444182e0a168d483940e (127.6.209.193:40299): tablet b36aa379d04a4f4db93270d302001054 (table test-table [id=01b093943376462d801873d02838ef60]) successfully deleted
I20260926 15:15:11.366844 8701 catalog_manager.cc:4841] TS 2cd15daf6de446359f72ef4769e57039 (127.6.209.194:33079): tablet b36aa379d04a4f4db93270d302001054 (table test-table [id=01b093943376462d801873d02838ef60]) successfully deleted
I20260926 15:15:11.369692 9317 tablet_bootstrap.cc:492] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039: No bootstrap required, opened a new log
I20260926 15:15:11.369853 9317 ts_tablet_manager.cc:1397] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.005s
I20260926 15:15:11.370250 9315 tablet_bootstrap.cc:492] T 9b3328532c244f4ab687bb3ef1d23b5c P c2f9aa3153fc487c8c5cdb06e521ba8f: No bootstrap required, opened a new log
I20260926 15:15:11.370428 9315 ts_tablet_manager.cc:1397] T 9b3328532c244f4ab687bb3ef1d23b5c P c2f9aa3153fc487c8c5cdb06e521ba8f: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.002s
I20260926 15:15:11.370491 9317 raft_consensus.cc:348] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:11.370698 9317 raft_consensus.cc:374] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:11.370762 9317 raft_consensus.cc:729] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd15daf6de446359f72ef4769e57039, State: Initialized, Role: FOLLOWER
I20260926 15:15:11.370958 9317 consensus_queue.cc:260] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:11.371125 9315 raft_consensus.cc:348] T 9b3328532c244f4ab687bb3ef1d23b5c P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:11.371349 9315 raft_consensus.cc:374] T 9b3328532c244f4ab687bb3ef1d23b5c P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:11.371398 9317 ts_tablet_manager.cc:1428] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 15:15:11.371443 9315 raft_consensus.cc:729] T 9b3328532c244f4ab687bb3ef1d23b5c P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2f9aa3153fc487c8c5cdb06e521ba8f, State: Initialized, Role: FOLLOWER
I20260926 15:15:11.371697 9315 consensus_queue.cc:260] T 9b3328532c244f4ab687bb3ef1d23b5c P c2f9aa3153fc487c8c5cdb06e521ba8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:11.372169 9315 ts_tablet_manager.cc:1428] T 9b3328532c244f4ab687bb3ef1d23b5c P c2f9aa3153fc487c8c5cdb06e521ba8f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 15:15:11.404937 9320 raft_consensus.cc:482] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:15:11.405139 9320 raft_consensus.cc:504] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:11.406834 9320 leader_election.cc:288] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9caa979a9710444182e0a168d483940e (127.6.209.193:40299), c2f9aa3153fc487c8c5cdb06e521ba8f (127.6.209.195:35275)
I20260926 15:15:11.413264 9334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:11.407088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:40299 (local address 127.6.209.194:32817)
0926 15:15:11.407714 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:11.407733 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:11.407845 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:11.408366 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:11.408368 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:11.408386 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:11.409173 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:11.409187 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.410445 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.410453 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:11.411620 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:11.411630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.412601 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.412606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:11.412754 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:11.412786 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 15:15:11.412886 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 15:15:11.412992 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":412,"threads_started":1}
I20260926 15:15:11.413283 9333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:11.407060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:32817 (local address 127.6.209.193:40299)
0926 15:15:11.407567 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:11.407572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:11.407946 (+ 374us) server_negotiation.cc:407] Connection header received
0926 15:15:11.408011 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:11.408017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:11.408098 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:11.408192 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:11.409324 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.410316 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.411751 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.412446 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.412601 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:11.412668 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:11.412906 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 15:15:11.413023 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":218,"threads_started":1}
I20260926 15:15:11.414182 8878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3328532c244f4ab687bb3ef1d23b5c" candidate_uuid: "2cd15daf6de446359f72ef4769e57039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9caa979a9710444182e0a168d483940e" is_pre_election: true
I20260926 15:15:11.414392 8878 raft_consensus.cc:2454] T 9b3328532c244f4ab687bb3ef1d23b5c P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cd15daf6de446359f72ef4769e57039 in term 0.
I20260926 15:15:11.414475 9323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:11.407943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:40855 (local address 127.6.209.195:35275)
0926 15:15:11.408073 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:11.408078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:11.408616 (+ 538us) server_negotiation.cc:407] Connection header received
0926 15:15:11.409163 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:11.409167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:11.409257 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:11.409348 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:11.409747 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.410396 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.412464 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.413494 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.413701 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:11.413765 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:11.414089 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 15:15:11.414207 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 15:15:11.415066 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:11.407848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:35275 (local address 127.6.209.194:40855)
0926 15:15:11.408516 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:11.408536 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:11.408674 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:11.409359 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:11.409363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:11.409383 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:11.409619 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:11.409626 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.410494 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.410499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:11.412323 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:11.412336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.413725 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.413731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:11.413897 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:11.413932 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 15:15:11.414589 (+ 657us) client_negotiation.cc:240] Negotiation successful
0926 15:15:11.414714 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":337,"threads_started":1}
I20260926 15:15:11.415040 8946 leader_election.cc:302] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cd15daf6de446359f72ef4769e57039, 9caa979a9710444182e0a168d483940e; no voters:
I20260926 15:15:11.415357 9320 raft_consensus.cc:2781] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:15:11.415472 9320 raft_consensus.cc:482] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:15:11.415556 9320 raft_consensus.cc:3037] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:11.415720 9151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3328532c244f4ab687bb3ef1d23b5c" candidate_uuid: "2cd15daf6de446359f72ef4769e57039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" is_pre_election: true
I20260926 15:15:11.415917 9151 raft_consensus.cc:2454] T 9b3328532c244f4ab687bb3ef1d23b5c P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cd15daf6de446359f72ef4769e57039 in term 0.
I20260926 15:15:11.418141 9320 raft_consensus.cc:504] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:11.418606 9320 leader_election.cc:288] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039 [CANDIDATE]: Term 1 election: Requested vote from peers 9caa979a9710444182e0a168d483940e (127.6.209.193:40299), c2f9aa3153fc487c8c5cdb06e521ba8f (127.6.209.195:35275)
I20260926 15:15:11.419028 9151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3328532c244f4ab687bb3ef1d23b5c" candidate_uuid: "2cd15daf6de446359f72ef4769e57039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f"
I20260926 15:15:11.419031 8878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3328532c244f4ab687bb3ef1d23b5c" candidate_uuid: "2cd15daf6de446359f72ef4769e57039" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9caa979a9710444182e0a168d483940e"
I20260926 15:15:11.419190 9151 raft_consensus.cc:3037] T 9b3328532c244f4ab687bb3ef1d23b5c P c2f9aa3153fc487c8c5cdb06e521ba8f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:11.419186 8878 raft_consensus.cc:3037] T 9b3328532c244f4ab687bb3ef1d23b5c P 9caa979a9710444182e0a168d483940e [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:11.421023 9151 raft_consensus.cc:2454] T 9b3328532c244f4ab687bb3ef1d23b5c P c2f9aa3153fc487c8c5cdb06e521ba8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cd15daf6de446359f72ef4769e57039 in term 1.
I20260926 15:15:11.421396 8946 leader_election.cc:302] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cd15daf6de446359f72ef4769e57039, c2f9aa3153fc487c8c5cdb06e521ba8f; no voters:
I20260926 15:15:11.421628 9320 raft_consensus.cc:2781] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:11.421830 8878 raft_consensus.cc:2454] T 9b3328532c244f4ab687bb3ef1d23b5c P 9caa979a9710444182e0a168d483940e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cd15daf6de446359f72ef4769e57039 in term 1.
I20260926 15:15:11.422153 9320 raft_consensus.cc:686] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039 [term 1 LEADER]: Becoming Leader. State: Replica: 2cd15daf6de446359f72ef4769e57039, State: Running, Role: LEADER
I20260926 15:15:11.422529 9320 consensus_queue.cc:237] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } }
I20260926 15:15:11.425289 8717 catalog_manager.cc:5495] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cd15daf6de446359f72ef4769e57039 (127.6.209.194). New cstate: current_term: 1 leader_uuid: "2cd15daf6de446359f72ef4769e57039" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cd15daf6de446359f72ef4769e57039" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 } health_report { overall_health: UNKNOWN } } }
I20260926 15:15:11.447952 9338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:11.440959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36256 (local address 127.6.209.194:33079)
0926 15:15:11.441923 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:11.441929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:11.441951 (+ 22us) server_negotiation.cc:407] Connection header received
0926 15:15:11.442034 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:11.442039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:11.442146 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:11.442303 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:15:11.442719 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.443834 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.445096 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.446118 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.446239 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:11.446624 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:15:11.447369 (+ 745us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:15:11.447524 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:11.447573 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 15:15:11.447639 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":829,"thread_start_us":701,"threads_started":1}
I20260926 15:15:11.463945 9323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:11.457511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:56701 (local address 127.6.209.195:35275)
0926 15:15:11.457684 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:11.457690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:11.457977 (+ 287us) server_negotiation.cc:407] Connection header received
0926 15:15:11.458122 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:11.458126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:11.458195 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:11.458289 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:15:11.459291 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.460096 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.462097 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.462700 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.462824 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:11.463308 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:15:11.463447 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:15:11.463536 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:11.463673 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 15:15:11.463735 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 15:15:11.464129 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:11.457210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:35275 (local address 127.6.209.194:56701)
0926 15:15:11.457854 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:11.457873 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:11.458003 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:11.458318 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:11.458323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:11.458336 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:15:11.459089 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:11.459100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.460234 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.460239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:11.461943 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:11.461956 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.462884 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.462891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:11.463075 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:11.463135 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:15:11.463556 (+ 421us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:15:11.463566 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 15:15:11.463647 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 15:15:11.463779 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":352,"threads_started":1}
I20260926 15:15:11.472378 9151 raft_consensus.cc:1261] T 9b3328532c244f4ab687bb3ef1d23b5c P c2f9aa3153fc487c8c5cdb06e521ba8f [term 1 FOLLOWER]: Refusing update from remote peer 2cd15daf6de446359f72ef4769e57039: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:11.472364 8878 raft_consensus.cc:1261] T 9b3328532c244f4ab687bb3ef1d23b5c P 9caa979a9710444182e0a168d483940e [term 1 FOLLOWER]: Refusing update from remote peer 2cd15daf6de446359f72ef4769e57039: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:11.473116 9336 consensus_queue.cc:1035] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9caa979a9710444182e0a168d483940e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:11.473474 9320 consensus_queue.cc:1035] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2f9aa3153fc487c8c5cdb06e521ba8f" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:11.504309 9326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:11.497696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:33079 (local address 127.6.209.195:41109)
0926 15:15:11.498043 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:11.498062 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:11.498162 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:11.498782 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:11.498790 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:11.498807 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:15:11.499086 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:11.499108 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.500697 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.500703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:11.502354 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:11.502369 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.503100 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.503107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:11.503292 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:11.503340 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:15:11.503699 (+ 359us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:15:11.503711 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 15:15:11.503808 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 15:15:11.503954 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217}
I20260926 15:15:11.504304 9338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:11.498349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:41109 (local address 127.6.209.194:33079)
0926 15:15:11.498516 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:11.498522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:11.498539 (+ 17us) server_negotiation.cc:407] Connection header received
0926 15:15:11.498593 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:11.498597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:11.498665 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:11.498748 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:15:11.499442 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.500533 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.502514 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:11.502937 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:11.503039 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:11.503487 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:15:11.503583 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:15:11.503662 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:11.503864 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 15:15:11.503980 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 15:15:11.505787 9284 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 15:15:13.096781 8715 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:13.097075 8715 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:13.101141 8715 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a9b105f7a9749c985c9d52d0f08e7b1: Sending DeleteTablet for 3 replicas of tablet 9b3328532c244f4ab687bb3ef1d23b5c
I20260926 15:15:13.101794 8857 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b3328532c244f4ab687bb3ef1d23b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:13 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:57308
I20260926 15:15:13.102005 8991 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b3328532c244f4ab687bb3ef1d23b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:13 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:36248
I20260926 15:15:13.102696 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 8788
I20260926 15:15:13.105366 9130 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b3328532c244f4ab687bb3ef1d23b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:13 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:40046
I20260926 15:15:13.110049 9367 tablet_replica.cc:330] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039: stopping tablet replica
I20260926 15:15:13.110246 9367 raft_consensus.cc:2229] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:15:13.110664 9367 raft_consensus.cc:2258] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:13.111362 9370 tablet_replica.cc:330] T 9b3328532c244f4ab687bb3ef1d23b5c P c2f9aa3153fc487c8c5cdb06e521ba8f: stopping tablet replica
I20260926 15:15:13.111534 9370 raft_consensus.cc:2229] T 9b3328532c244f4ab687bb3ef1d23b5c P c2f9aa3153fc487c8c5cdb06e521ba8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:13.111732 9370 raft_consensus.cc:2258] T 9b3328532c244f4ab687bb3ef1d23b5c P c2f9aa3153fc487c8c5cdb06e521ba8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:13.111917 9367 ts_tablet_manager.cc:1905] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:13.112660 9370 ts_tablet_manager.cc:1905] T 9b3328532c244f4ab687bb3ef1d23b5c P c2f9aa3153fc487c8c5cdb06e521ba8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:13.117873 9366 tablet_replica.cc:330] T 9b3328532c244f4ab687bb3ef1d23b5c P 9caa979a9710444182e0a168d483940e: stopping tablet replica
I20260926 15:15:13.118040 9366 raft_consensus.cc:2229] T 9b3328532c244f4ab687bb3ef1d23b5c P 9caa979a9710444182e0a168d483940e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:13.118216 9366 raft_consensus.cc:2258] T 9b3328532c244f4ab687bb3ef1d23b5c P 9caa979a9710444182e0a168d483940e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:13.118515 9367 ts_tablet_manager.cc:1918] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 15:15:13.118680 9367 log.cc:1196] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/9b3328532c244f4ab687bb3ef1d23b5c
I20260926 15:15:13.119136 9367 ts_tablet_manager.cc:1939] T 9b3328532c244f4ab687bb3ef1d23b5c P 2cd15daf6de446359f72ef4769e57039: Deleting consensus metadata
I20260926 15:15:13.119383 9366 ts_tablet_manager.cc:1905] T 9b3328532c244f4ab687bb3ef1d23b5c P 9caa979a9710444182e0a168d483940e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:13.119762 9370 ts_tablet_manager.cc:1918] T 9b3328532c244f4ab687bb3ef1d23b5c P c2f9aa3153fc487c8c5cdb06e521ba8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 15:15:13.119933 9370 log.cc:1196] T 9b3328532c244f4ab687bb3ef1d23b5c P c2f9aa3153fc487c8c5cdb06e521ba8f: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/9b3328532c244f4ab687bb3ef1d23b5c
I20260926 15:15:13.120149 8701 catalog_manager.cc:4841] TS 2cd15daf6de446359f72ef4769e57039 (127.6.209.194:33079): tablet 9b3328532c244f4ab687bb3ef1d23b5c (table test-table [id=cf214486f57744228c777f2b82f83b08]) successfully deleted
I20260926 15:15:13.120383 9370 ts_tablet_manager.cc:1939] T 9b3328532c244f4ab687bb3ef1d23b5c P c2f9aa3153fc487c8c5cdb06e521ba8f: Deleting consensus metadata
I20260926 15:15:13.121388 8704 catalog_manager.cc:4841] TS c2f9aa3153fc487c8c5cdb06e521ba8f (127.6.209.195:35275): tablet 9b3328532c244f4ab687bb3ef1d23b5c (table test-table [id=cf214486f57744228c777f2b82f83b08]) successfully deleted
Sep 26 15:15:13 dist-test-slave-j6q7 krb5kdc[8672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.209.193@KRBTEST.COM
I20260926 15:15:13.124920 9366 ts_tablet_manager.cc:1918] T 9b3328532c244f4ab687bb3ef1d23b5c P 9caa979a9710444182e0a168d483940e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 15:15:13.125077 9366 log.cc:1196] T 9b3328532c244f4ab687bb3ef1d23b5c P 9caa979a9710444182e0a168d483940e: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435694426955-6983-0/minicluster-data/ts-0/wal/wals/9b3328532c244f4ab687bb3ef1d23b5c
I20260926 15:15:13.126644 9366 ts_tablet_manager.cc:1939] T 9b3328532c244f4ab687bb3ef1d23b5c P 9caa979a9710444182e0a168d483940e: Deleting consensus metadata
I20260926 15:15:13.127641 8704 catalog_manager.cc:4841] TS 9caa979a9710444182e0a168d483940e (127.6.209.193:40299): tablet 9b3328532c244f4ab687bb3ef1d23b5c (table test-table [id=cf214486f57744228c777f2b82f83b08]) successfully deleted
I20260926 15:15:13.129629 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:13.104241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56376 (local address 127.6.209.193:40299)
0926 15:15:13.112202 (+ 7961us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:13.112207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:13.112728 (+ 521us) server_negotiation.cc:407] Connection header received
0926 15:15:13.112777 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:13.112784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:13.112873 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:13.112955 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:13.114572 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:13.115582 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:13.118626 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:13.119266 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:13.119375 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:13.123832 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:13.123848 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:13.123867 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:13.123901 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:13.127417 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:13.128045 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:13.128052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:13.128054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:13.128161 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:13.128437 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:13.128439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:13.128441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:13.128713 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:13.128978 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:13.129223 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 15:15:13.129393 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7843,"thread_start_us":7762,"threads_started":1}
W20260926 15:15:13.130307 8918 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56376
W20260926 15:15:13.130843 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 8788: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:15:13.131039 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 8788
I20260926 15:15:13.151314 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 8927
Sep 26 15:15:13 dist-test-slave-j6q7 krb5kdc[8672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.209.194@KRBTEST.COM
I20260926 15:15:13.164693 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:13.151933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54626 (local address 127.6.209.194:33079)
0926 15:15:13.152373 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:13.152377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:13.152390 (+ 13us) server_negotiation.cc:407] Connection header received
0926 15:15:13.152430 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:13.152433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:13.152481 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:13.152616 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:13.154105 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:13.154807 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:13.155958 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:13.156332 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:13.156456 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:13.159750 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:13.159760 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:13.159773 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:13.159797 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:13.162307 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:13.162838 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:13.162849 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:13.162851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:13.162961 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:13.163268 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:13.163272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:13.163275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:13.163603 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:13.163877 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:13.164158 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 15:15:13.164362 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":235,"threads_started":1}
W20260926 15:15:13.165285 9057 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54626
W20260926 15:15:13.165863 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 8927: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:15:13.165980 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 8927
I20260926 15:15:13.185434 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 9066
Sep 26 15:15:13 dist-test-slave-j6q7 krb5kdc[8672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.209.195@KRBTEST.COM
I20260926 15:15:13.197026 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:13.185922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38430 (local address 127.6.209.195:35275)
0926 15:15:13.186351 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:13.186354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:13.186367 (+ 13us) server_negotiation.cc:407] Connection header received
0926 15:15:13.186406 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:13.186409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:13.186458 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:13.186552 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:13.187489 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:13.188088 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:13.189127 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:13.189446 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:13.189527 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:13.192647 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:13.192658 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:13.192671 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:13.192695 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:13.195032 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:13.195490 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:13.195496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:13.195498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:13.195597 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:13.195862 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:13.195864 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:13.195866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:13.196118 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:13.196320 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:13.196481 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 15:15:13.196620 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":229,"threads_started":1}
W20260926 15:15:13.197533 9196 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38430
W20260926 15:15:13.197960 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 9066: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:15:13.198101 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 9066
I20260926 15:15:13.220852 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 8685
I20260926 15:15:13.221783 8757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:13.326475 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 8685
2026-09-26T15:15:13Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7776 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:15:13 dist-test-slave-j6q7 krb5kdc[9376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:15:13 dist-test-slave-j6q7 krb5kdc[9376](info): set up 2 sockets
Sep 26 15:15:13 dist-test-slave-j6q7 krb5kdc[9376](info): commencing operation
krb5kdc: starting...
W20260926 15:15:14.316706 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:15:14 dist-test-slave-j6q7 krb5kdc[9376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:15:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:15:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:15:14.473838 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:35171
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:46283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:35171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:14.765182 9389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:14.765437 9389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:14.765528 9389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:14.771795 9389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:15:14.771880 9389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:14.771931 9389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:14.771984 9389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:15:14.772025 9389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:15:14.780300 9389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:46283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:35171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:35171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:14.781394 9389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:14.782509 9389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:14.793164 9397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:14.793279 9394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:14.793256 9395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:14.793982 9389 server_base.cc:1034] running on GCE node
I20260926 15:15:14.794636 9389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:14.795895 9389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:14.797087 9389 hybrid_clock.cc:648] HybridClock initialized: now 1790435714797060 us; error 79 us; skew 500 ppm
I20260926 15:15:14.797381 9389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:14 dist-test-slave-j6q7 krb5kdc[9376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435714, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:14.799947 9389 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:15:14.801151 9389 webserver.cc:460] Webserver started at http://127.6.209.254:46653/ using document root <none> and password file <none>
I20260926 15:15:14.801697 9389 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:14.801783 9389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:14.802004 9389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:14.804453 9389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "50758b4fa2e04f01a048108f118ed412"
format_stamp: "Formatted at 2026-09-26 15:15:14 on dist-test-slave-j6q7"
server_key: "3a8f5808452d1c749ea5c87606f8a24c"
server_key_iv: "7ddaa56c00364bef897bc96391bb78d0"
server_key_version: "encryptionkey@0"
I20260926 15:15:14.805186 9389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "50758b4fa2e04f01a048108f118ed412"
format_stamp: "Formatted at 2026-09-26 15:15:14 on dist-test-slave-j6q7"
server_key: "3a8f5808452d1c749ea5c87606f8a24c"
server_key_iv: "7ddaa56c00364bef897bc96391bb78d0"
server_key_version: "encryptionkey@0"
I20260926 15:15:14.809398 9389 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:15:14.812237 9404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:14.813354 9389 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:15:14.813524 9389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "50758b4fa2e04f01a048108f118ed412"
format_stamp: "Formatted at 2026-09-26 15:15:14 on dist-test-slave-j6q7"
server_key: "3a8f5808452d1c749ea5c87606f8a24c"
server_key_iv: "7ddaa56c00364bef897bc96391bb78d0"
server_key_version: "encryptionkey@0"
I20260926 15:15:14.813735 9389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:14.829411 9389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:14.833006 9389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:14.833262 9389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:14.848902 9389 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:35171
I20260926 15:15:14.848877 9455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:35171 every 8 connection(s)
I20260926 15:15:14.850015 9389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:15:14.853127 9456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:14.857227 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 9389
I20260926 15:15:14.857432 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:15:14.857931 6983 external_mini_cluster.cc:1428] Setting key 10a572226f07365eb48fe25c2cd28866
I20260926 15:15:14.862491 9456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50758b4fa2e04f01a048108f118ed412: Bootstrap starting.
I20260926 15:15:14.865836 9456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50758b4fa2e04f01a048108f118ed412: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:14.866920 9456 log.cc:824] T 00000000000000000000000000000000 P 50758b4fa2e04f01a048108f118ed412: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:14.869901 9456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50758b4fa2e04f01a048108f118ed412: No bootstrap required, opened a new log
Sep 26 15:15:14 dist-test-slave-j6q7 krb5kdc[9376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:14.879381 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:14.859512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34284 (local address 127.6.209.254:35171)
0926 15:15:14.860201 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:14.860211 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:14.860243 (+ 32us) server_negotiation.cc:407] Connection header received
0926 15:15:14.861833 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:14.861853 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:14.862187 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:14.862485 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:14.864207 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:14.865453 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:14.867159 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:14.868125 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:14.868247 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:14.872188 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:14.872198 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:14.872217 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:14.872259 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:14.875802 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:14.876335 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:14.876344 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:14.876352 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:14.876552 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:14.876833 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:14.876839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:14.876845 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:14.877418 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:14.877689 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:14.877973 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 15:15:14.878349 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":323,"threads_started":1}
I20260926 15:15:14.879182 9456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50758b4fa2e04f01a048108f118ed412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50758b4fa2e04f01a048108f118ed412" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35171 } }
I20260926 15:15:14.879573 9456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50758b4fa2e04f01a048108f118ed412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:14.879666 9456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50758b4fa2e04f01a048108f118ed412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50758b4fa2e04f01a048108f118ed412, State: Initialized, Role: FOLLOWER
I20260926 15:15:14.880121 9456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50758b4fa2e04f01a048108f118ed412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50758b4fa2e04f01a048108f118ed412" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35171 } }
I20260926 15:15:14.880321 9456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50758b4fa2e04f01a048108f118ed412 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:15:14.880432 9456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50758b4fa2e04f01a048108f118ed412 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:15:14.880568 9456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50758b4fa2e04f01a048108f118ed412 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:14.882673 9456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50758b4fa2e04f01a048108f118ed412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50758b4fa2e04f01a048108f118ed412" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35171 } }
I20260926 15:15:14.883103 9456 leader_election.cc:302] T 00000000000000000000000000000000 P 50758b4fa2e04f01a048108f118ed412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50758b4fa2e04f01a048108f118ed412; no voters:
I20260926 15:15:14.883592 9456 leader_election.cc:288] T 00000000000000000000000000000000 P 50758b4fa2e04f01a048108f118ed412 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:15:14.883730 9461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50758b4fa2e04f01a048108f118ed412 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:14.884419 9461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50758b4fa2e04f01a048108f118ed412 [term 1 LEADER]: Becoming Leader. State: Replica: 50758b4fa2e04f01a048108f118ed412, State: Running, Role: LEADER
I20260926 15:15:14.884964 9461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50758b4fa2e04f01a048108f118ed412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50758b4fa2e04f01a048108f118ed412" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35171 } }
I20260926 15:15:14.885471 9456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50758b4fa2e04f01a048108f118ed412 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:15:14.888190 9463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50758b4fa2e04f01a048108f118ed412 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50758b4fa2e04f01a048108f118ed412. Latest consensus state: current_term: 1 leader_uuid: "50758b4fa2e04f01a048108f118ed412" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50758b4fa2e04f01a048108f118ed412" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35171 } } }
I20260926 15:15:14.888273 9462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50758b4fa2e04f01a048108f118ed412 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50758b4fa2e04f01a048108f118ed412" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50758b4fa2e04f01a048108f118ed412" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35171 } } }
I20260926 15:15:14.888682 9463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50758b4fa2e04f01a048108f118ed412 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:14.888679 9462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50758b4fa2e04f01a048108f118ed412 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:14.889467 9468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:15:14.903712 9468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:15:14.911020 9468 catalog_manager.cc:1347] Generated new cluster ID: 6887f8c4a2704db18cee7626196f5a48
I20260926 15:15:14.911118 9468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:15:14.926121 9468 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:15:14.927191 9468 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:15:14.934377 9468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50758b4fa2e04f01a048108f118ed412: Generated new TSK 0
I20260926 15:15:14.935042 9468 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
I20260926 15:15:14.988535 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35171
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:46283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:15.286806 9482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:15.287060 9482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:15.287205 9482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:15.293215 9482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:15.293322 9482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:15.293466 9482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:15:15.301635 9482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:46283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:15.302742 9482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:15.303810 9482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:15.315157 9490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:15.315253 9487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:15.315229 9488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:15.315944 9482 server_base.cc:1034] running on GCE node
I20260926 15:15:15.316419 9482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:15.317309 9482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:15.318508 9482 hybrid_clock.cc:648] HybridClock initialized: now 1790435715318476 us; error 43 us; skew 500 ppm
I20260926 15:15:15.318763 9482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:15 dist-test-slave-j6q7 krb5kdc[9376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435715, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:15.321470 9482 init.cc:377] Logged in from keytab as kudu/127.6.209.193@KRBTEST.COM (short username kudu)
I20260926 15:15:15.322806 9482 webserver.cc:460] Webserver started at http://127.6.209.193:36991/ using document root <none> and password file <none>
I20260926 15:15:15.323457 9482 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:15.323583 9482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:15.323843 9482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:15.326535 9482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "ef96c7bda54742ff9e80d1b1da94e83c"
format_stamp: "Formatted at 2026-09-26 15:15:15 on dist-test-slave-j6q7"
server_key: "b4e9312404f7a4e3288bf48f548ecb5d"
server_key_iv: "d7cc8dcbf8ae51b723ce936dead3cb91"
server_key_version: "encryptionkey@0"
I20260926 15:15:15.327291 9482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "ef96c7bda54742ff9e80d1b1da94e83c"
format_stamp: "Formatted at 2026-09-26 15:15:15 on dist-test-slave-j6q7"
server_key: "b4e9312404f7a4e3288bf48f548ecb5d"
server_key_iv: "d7cc8dcbf8ae51b723ce936dead3cb91"
server_key_version: "encryptionkey@0"
I20260926 15:15:15.331517 9482 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 15:15:15.334338 9497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:15.335471 9482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 15:15:15.335619 9482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "ef96c7bda54742ff9e80d1b1da94e83c"
format_stamp: "Formatted at 2026-09-26 15:15:15 on dist-test-slave-j6q7"
server_key: "b4e9312404f7a4e3288bf48f548ecb5d"
server_key_iv: "d7cc8dcbf8ae51b723ce936dead3cb91"
server_key_version: "encryptionkey@0"
I20260926 15:15:15.335777 9482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:15.351725 9482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:15.355118 9482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:15.355396 9482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:15.356103 9482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:15.357188 9482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:15.357275 9482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:15.357416 9482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:15.357501 9482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:15.386029 9482 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:39571
I20260926 15:15:15.386097 9609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:39571 every 8 connection(s)
I20260926 15:15:15.387212 9482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:15:15.394079 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 9482
I20260926 15:15:15.394281 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:15:15.394887 6983 external_mini_cluster.cc:1428] Setting key 9ec31b0e2edd8ec902a1dea57ea4e177
Sep 26 15:15:15 dist-test-slave-j6q7 krb5kdc[9376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435715, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:15.405284 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:15.389319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:41703 (local address 127.6.209.254:35171)
0926 15:15:15.389639 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:15.389644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:15.390467 (+ 823us) server_negotiation.cc:407] Connection header received
0926 15:15:15.391868 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:15.391871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:15.391925 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:15.392011 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:15.393266 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:15.394161 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:15.395736 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:15.396090 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:15.396170 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:15.399275 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:15.399280 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:15.399283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:15.399310 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:15.402327 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:15.402948 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:15.402955 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:15.402961 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:15.403075 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:15.403447 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:15.403454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:15.403457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:15.403697 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:15.403838 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:15.404589 (+ 751us) server_negotiation.cc:299] Negotiation successful
0926 15:15:15.404851 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":164,"threads_started":1}
I20260926 15:15:15.406052 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:15.389800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:35171 (local address 127.6.209.193:41703)
0926 15:15:15.390342 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:15.390372 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:15.391663 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:15.392171 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:15.392179 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:15.392623 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:15.393141 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:15.393156 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:15.394302 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:15.394305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:15.395630 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:15.395640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:15.396276 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:15.396289 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:15.396531 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:15.397168 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:15.397197 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:15.399072 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:15.402450 (+ 3378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:15.402457 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:15.402469 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:15.402838 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:15.403176 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:15.403182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:15.403186 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:15.403345 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:15.403813 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:15.403820 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:15.404127 (+ 307us) client_negotiation.cc:769] Sending connection context
0926 15:15:15.404476 (+ 349us) client_negotiation.cc:240] Negotiation successful
0926 15:15:15.404985 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":210,"threads_started":1}
I20260926 15:15:15.407387 9610 heartbeater.cc:347] Connected to a master server at 127.6.209.254:35171
I20260926 15:15:15.407647 9610 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:15.408264 9610 heartbeater.cc:511] Master 127.6.209.254:35171 requested a full tablet report, sending...
I20260926 15:15:15.410229 9421 ts_manager.cc:194] Registered new tserver with Master: ef96c7bda54742ff9e80d1b1da94e83c (127.6.209.193:39571)
WARNING: no policy specified for kudu/127.6.209.194@KRBTEST.COM; defaulting to no policy
I20260926 15:15:15.411631 9421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.193@KRBTEST.COM'} at 127.6.209.193:41703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
I20260926 15:15:15.432291 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35171
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:46283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:15.726281 9616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:15.726552 9616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:15.726704 9616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:15.732882 9616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:15.732999 9616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:15.733142 9616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:15:15.741639 9616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:46283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:15.742671 9616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:15.743813 9616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:15.755323 9621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:15.755380 9624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:15.755403 9622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:15.756085 9616 server_base.cc:1034] running on GCE node
I20260926 15:15:15.756585 9616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:15.757330 9616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:15.758502 9616 hybrid_clock.cc:648] HybridClock initialized: now 1790435715758470 us; error 26 us; skew 500 ppm
I20260926 15:15:15.758772 9616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:15 dist-test-slave-j6q7 krb5kdc[9376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435715, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:15.761389 9616 init.cc:377] Logged in from keytab as kudu/127.6.209.194@KRBTEST.COM (short username kudu)
I20260926 15:15:15.762678 9616 webserver.cc:460] Webserver started at http://127.6.209.194:46327/ using document root <none> and password file <none>
I20260926 15:15:15.763235 9616 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:15.763358 9616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:15.763616 9616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:15.766183 9616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "daa686621c2644f79fcf1518cf69189b"
format_stamp: "Formatted at 2026-09-26 15:15:15 on dist-test-slave-j6q7"
server_key: "9df2ec905bd6d5dfafa857fb4045ebe9"
server_key_iv: "7ca88b66db133e07639a6aa0b7da7d38"
server_key_version: "encryptionkey@0"
I20260926 15:15:15.766857 9616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "daa686621c2644f79fcf1518cf69189b"
format_stamp: "Formatted at 2026-09-26 15:15:15 on dist-test-slave-j6q7"
server_key: "9df2ec905bd6d5dfafa857fb4045ebe9"
server_key_iv: "7ca88b66db133e07639a6aa0b7da7d38"
server_key_version: "encryptionkey@0"
I20260926 15:15:15.771116 9616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 15:15:15.773936 9631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:15.775141 9616 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:15:15.775281 9616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "daa686621c2644f79fcf1518cf69189b"
format_stamp: "Formatted at 2026-09-26 15:15:15 on dist-test-slave-j6q7"
server_key: "9df2ec905bd6d5dfafa857fb4045ebe9"
server_key_iv: "7ca88b66db133e07639a6aa0b7da7d38"
server_key_version: "encryptionkey@0"
I20260926 15:15:15.775444 9616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:15.788146 9616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:15.791348 9616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:15.791584 9616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:15.792413 9616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:15.793632 9616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:15.793740 9616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:15.793869 9616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:15.793936 9616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:15.824896 9616 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:40633
I20260926 15:15:15.824944 9743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:40633 every 8 connection(s)
I20260926 15:15:15.826006 9616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:15:15.835937 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 9616
I20260926 15:15:15.836104 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:15:15.836477 6983 external_mini_cluster.cc:1428] Setting key b7d8c6ba71fcfff585827dd16a6fc1c3
Sep 26 15:15:15 dist-test-slave-j6q7 krb5kdc[9376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435715, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:15.844409 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:15.828151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:47255 (local address 127.6.209.254:35171)
0926 15:15:15.828310 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:15.828314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:15.829353 (+ 1039us) server_negotiation.cc:407] Connection header received
0926 15:15:15.830896 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:15.830901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:15.830983 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:15.831085 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:15.832343 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:15.833088 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:15.834577 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:15.834990 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:15.835119 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:15.838390 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:15.838394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:15.838398 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:15.838431 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:15.841642 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:15.842245 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:15.842253 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:15.842258 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:15.842400 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:15.842760 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:15.842766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:15.842772 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:15.843111 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:15.843247 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:15.843761 (+ 514us) server_negotiation.cc:299] Negotiation successful
0926 15:15:15.843979 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 15:15:15.844966 9746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:15.828563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:35171 (local address 127.6.209.194:47255)
0926 15:15:15.829214 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:15.829252 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:15.830668 (+ 1416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:15.831283 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:15.831289 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:15.831725 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:15.832184 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:15.832198 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:15.833246 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:15.833250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:15.834420 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:15.834431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:15.835203 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:15.835209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:15.835395 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:15.836021 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:15.836062 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:15.838232 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:15.841759 (+ 3527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:15.841765 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:15.841775 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:15.842137 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:15.842507 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:15.842510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:15.842512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:15.842657 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:15.843215 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:15.843219 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:15.843463 (+ 244us) client_negotiation.cc:769] Sending connection context
0926 15:15:15.843652 (+ 189us) client_negotiation.cc:240] Negotiation successful
0926 15:15:15.843941 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":283,"threads_started":1}
I20260926 15:15:15.846150 9744 heartbeater.cc:347] Connected to a master server at 127.6.209.254:35171
I20260926 15:15:15.846410 9744 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:15.847114 9744 heartbeater.cc:511] Master 127.6.209.254:35171 requested a full tablet report, sending...
I20260926 15:15:15.848407 9421 ts_manager.cc:194] Registered new tserver with Master: daa686621c2644f79fcf1518cf69189b (127.6.209.194:40633)
I20260926 15:15:15.849210 9421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.194@KRBTEST.COM'} at 127.6.209.194:47255
WARNING: no policy specified for kudu/127.6.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
I20260926 15:15:15.879132 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35171
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:46283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:16.161223 9749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:16.161450 9749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:16.161590 9749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:16.167835 9749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:16.167940 9749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:16.168102 9749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:15:16.176222 9749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:46283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:16.177237 9749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:16.178341 9749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:16.190383 9754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:16.190361 9755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:16.190708 9757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:16.191252 9749 server_base.cc:1034] running on GCE node
I20260926 15:15:16.191776 9749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:16.192509 9749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:16.193699 9749 hybrid_clock.cc:648] HybridClock initialized: now 1790435716193680 us; error 129 us; skew 500 ppm
I20260926 15:15:16.193979 9749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:16 dist-test-slave-j6q7 krb5kdc[9376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435716, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:16.200757 9749 init.cc:377] Logged in from keytab as kudu/127.6.209.195@KRBTEST.COM (short username kudu)
I20260926 15:15:16.202114 9749 webserver.cc:460] Webserver started at http://127.6.209.195:42497/ using document root <none> and password file <none>
I20260926 15:15:16.202739 9749 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:16.202889 9749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:16.203162 9749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:16.205745 9749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "1c67b9d1917347a089f881100f627110"
format_stamp: "Formatted at 2026-09-26 15:15:16 on dist-test-slave-j6q7"
server_key: "2ae39288cde920d679fe6324a803aad4"
server_key_iv: "29ac3a14c2fd0dc3a50673c80cdc1f7d"
server_key_version: "encryptionkey@0"
I20260926 15:15:16.206404 9749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "1c67b9d1917347a089f881100f627110"
format_stamp: "Formatted at 2026-09-26 15:15:16 on dist-test-slave-j6q7"
server_key: "2ae39288cde920d679fe6324a803aad4"
server_key_iv: "29ac3a14c2fd0dc3a50673c80cdc1f7d"
server_key_version: "encryptionkey@0"
I20260926 15:15:16.210459 9749 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 15:15:16.213162 9764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:16.214224 9749 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:15:16.214388 9749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "1c67b9d1917347a089f881100f627110"
format_stamp: "Formatted at 2026-09-26 15:15:16 on dist-test-slave-j6q7"
server_key: "2ae39288cde920d679fe6324a803aad4"
server_key_iv: "29ac3a14c2fd0dc3a50673c80cdc1f7d"
server_key_version: "encryptionkey@0"
I20260926 15:15:16.214565 9749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:16.226517 9749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:16.229996 9749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:16.230237 9749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:16.231000 9749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:16.232085 9749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:16.232179 9749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:16.232311 9749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:16.232379 9749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:16.262147 9749 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:40793
I20260926 15:15:16.262221 9876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:40793 every 8 connection(s)
I20260926 15:15:16.263228 9749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:15:16.272379 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 9749
I20260926 15:15:16.272545 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:15:16.273041 6983 external_mini_cluster.cc:1428] Setting key 00c9b8a2e7c30afc53d4490e822980fe
Sep 26 15:15:16 dist-test-slave-j6q7 krb5kdc[9376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435716, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:16.279624 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:16.265437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:55895 (local address 127.6.209.254:35171)
0926 15:15:16.265632 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:16.265638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:16.266604 (+ 966us) server_negotiation.cc:407] Connection header received
0926 15:15:16.268012 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:16.268016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:16.268074 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:16.268152 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:16.269352 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:16.269950 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:16.271199 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:16.271584 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:16.271662 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:16.275184 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:16.275188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:16.275191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:16.275219 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:16.277416 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:16.277969 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:16.277973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:16.277975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:16.278072 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:16.278377 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:16.278380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:16.278382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:16.278622 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:16.278720 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:16.279227 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 15:15:16.279392 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 15:15:16.280372 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:16.265875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:35171 (local address 127.6.209.195:55895)
0926 15:15:16.266459 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:16.266501 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:16.267813 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:16.268312 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:16.268318 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:16.268728 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:16.269230 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:16.269244 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:16.270061 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:16.270064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:16.271094 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:16.271104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:16.271728 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:16.271732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:16.271885 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:16.272424 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:16.272449 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:16.275014 (+ 2565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:16.277521 (+ 2507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:16.277526 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:16.277534 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:16.277886 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:16.278153 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:16.278156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:16.278158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:16.278299 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:16.278718 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:16.278724 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:16.278982 (+ 258us) client_negotiation.cc:769] Sending connection context
0926 15:15:16.279169 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 15:15:16.279443 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":230,"threads_started":1}
I20260926 15:15:16.281598 9877 heartbeater.cc:347] Connected to a master server at 127.6.209.254:35171
I20260926 15:15:16.281813 9877 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:16.282382 9877 heartbeater.cc:511] Master 127.6.209.254:35171 requested a full tablet report, sending...
I20260926 15:15:16.283526 9421 ts_manager.cc:194] Registered new tserver with Master: 1c67b9d1917347a089f881100f627110 (127.6.209.195:40793)
I20260926 15:15:16.284237 9421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.195@KRBTEST.COM'} at 127.6.209.195:55895
I20260926 15:15:16.287529 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 15:15:16.312611 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:15:16.310711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34286 (local address 127.6.209.254:35171)
0926 15:15:16.310834 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:16.310838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:16.311246 (+ 408us) server_negotiation.cc:407] Connection header received
0926 15:15:16.311368 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:16.311371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:16.311420 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:16.311528 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:16.312392 (+ 864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34286: BlockingRecv error: recv got EOF from 127.0.0.1:34286 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 15:15:16.316095 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 9482
W20260926 15:15:16.318522 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 9482: Not authorized: Client connection negotiation failed: client connection to 127.6.209.193:39571: server requires authentication, but client does not have Kerberos credentials available
I20260926 15:15:16.318686 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 9482
I20260926 15:15:16.332489 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 9616
W20260926 15:15:16.334560 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 9616: Not authorized: Client connection negotiation failed: client connection to 127.6.209.194:40633: server requires authentication, but client does not have Kerberos credentials available
I20260926 15:15:16.334699 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 9616
I20260926 15:15:16.348340 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 9749
W20260926 15:15:16.350708 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 9749: Not authorized: Client connection negotiation failed: client connection to 127.6.209.195:40793: server requires authentication, but client does not have Kerberos credentials available
I20260926 15:15:16.350797 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 9749
I20260926 15:15:16.364488 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 9389
I20260926 15:15:16.365460 9451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:16.465644 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 9389
2026-09-26T15:15:16Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3137 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:15:18 dist-test-slave-j6q7 krb5kdc[9894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:15:18 dist-test-slave-j6q7 krb5kdc[9894](info): set up 2 sockets
Sep 26 15:15:18 dist-test-slave-j6q7 krb5kdc[9894](info): commencing operation
krb5kdc: starting...
W20260926 15:15:19.363731 6983 mini_kdc.cc:121] Time spent starting KDC: real 2.858s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 15:15:19.507578 6983 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.144s user 0.000s sys 0.004s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260926 15:15:19.637089 6983 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.129s user 0.000s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260926 15:15:19.797078 6983 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.160s user 0.000s sys 0.004s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/krb5kdc/test-user.keytab.
W20260926 15:15:20.793536 6983 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.996s user 0.000s sys 0.004s
Sep 26 15:15:20 dist-test-slave-j6q7 krb5kdc[9894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:15:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:15:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
W20260926 15:15:21.109494 6983 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.6.209.254: real 0.270s user 0.000s sys 0.007s
I20260926 15:15:21.110363 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:40101
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:40101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:21.393064 9907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:21.393415 9907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:21.393576 9907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:21.399999 9907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:15:21.400128 9907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:21.400226 9907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:21.400332 9907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:15:21.400418 9907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:15:21.408777 9907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:40101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:40101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:21.409814 9907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:21.410936 9907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:21.422065 9913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:21.422036 9915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:21.422523 9912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:21.422693 9907 server_base.cc:1034] running on GCE node
I20260926 15:15:21.423354 9907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:21.424511 9907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:21.425753 9907 hybrid_clock.cc:648] HybridClock initialized: now 1790435721425700 us; error 55 us; skew 500 ppm
I20260926 15:15:21.426069 9907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:21 dist-test-slave-j6q7 krb5kdc[9894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435721, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:21.435323 9907 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:15:21.436607 9907 webserver.cc:460] Webserver started at http://127.6.209.254:40733/ using document root <none> and password file <none>
I20260926 15:15:21.437208 9907 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:21.437295 9907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:21.437511 9907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:21.439877 9907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "2b4903da5afb4ba691b22a68307a3480"
format_stamp: "Formatted at 2026-09-26 15:15:21 on dist-test-slave-j6q7"
server_key: "7dd212def27e58a87279699b5d1bb9d0"
server_key_iv: "68754d7fbf2d337b04154e9e9f8c47a8"
server_key_version: "encryptionkey@0"
I20260926 15:15:21.440513 9907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "2b4903da5afb4ba691b22a68307a3480"
format_stamp: "Formatted at 2026-09-26 15:15:21 on dist-test-slave-j6q7"
server_key: "7dd212def27e58a87279699b5d1bb9d0"
server_key_iv: "68754d7fbf2d337b04154e9e9f8c47a8"
server_key_version: "encryptionkey@0"
I20260926 15:15:21.444911 9907 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260926 15:15:21.447723 9922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:21.448784 9907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:15:21.448968 9907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "2b4903da5afb4ba691b22a68307a3480"
format_stamp: "Formatted at 2026-09-26 15:15:21 on dist-test-slave-j6q7"
server_key: "7dd212def27e58a87279699b5d1bb9d0"
server_key_iv: "68754d7fbf2d337b04154e9e9f8c47a8"
server_key_version: "encryptionkey@0"
I20260926 15:15:21.449123 9907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:21.477382 9907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:21.480482 9907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:21.480669 9907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:21.497057 9907 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:40101
I20260926 15:15:21.497123 9973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:40101 every 8 connection(s)
I20260926 15:15:21.498230 9907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:15:21.501578 9974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:21.505146 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 9907
I20260926 15:15:21.505355 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:15:21.505959 6983 external_mini_cluster.cc:1428] Setting key 57f838f4d8547282585343b1773193fa
I20260926 15:15:21.512542 9974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b4903da5afb4ba691b22a68307a3480: Bootstrap starting.
I20260926 15:15:21.515822 9974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b4903da5afb4ba691b22a68307a3480: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:21.516979 9974 log.cc:824] T 00000000000000000000000000000000 P 2b4903da5afb4ba691b22a68307a3480: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:21.519364 9974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b4903da5afb4ba691b22a68307a3480: No bootstrap required, opened a new log
Sep 26 15:15:21 dist-test-slave-j6q7 krb5kdc[9894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:21.528403 9977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:21.507548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38272 (local address 127.6.209.254:40101)
0926 15:15:21.508350 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:21.508365 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:21.508412 (+ 47us) server_negotiation.cc:407] Connection header received
0926 15:15:21.510992 (+ 2580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:21.511015 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:21.511405 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:21.511778 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:21.513027 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:21.514875 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:21.516086 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:21.517049 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:21.517212 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:21.520675 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:21.520689 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:21.520710 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:21.520750 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:21.524381 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:21.525073 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:21.525081 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:21.525089 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:21.525287 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:21.525616 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:21.525621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:21.525625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:21.526300 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:21.526579 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:21.526803 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 15:15:21.527169 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":342,"threads_started":1}
I20260926 15:15:21.529316 9974 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b4903da5afb4ba691b22a68307a3480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b4903da5afb4ba691b22a68307a3480" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40101 } }
I20260926 15:15:21.529623 9974 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b4903da5afb4ba691b22a68307a3480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:21.529737 9974 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b4903da5afb4ba691b22a68307a3480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b4903da5afb4ba691b22a68307a3480, State: Initialized, Role: FOLLOWER
I20260926 15:15:21.530227 9974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b4903da5afb4ba691b22a68307a3480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b4903da5afb4ba691b22a68307a3480" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40101 } }
I20260926 15:15:21.530439 9974 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b4903da5afb4ba691b22a68307a3480 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:15:21.530578 9974 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b4903da5afb4ba691b22a68307a3480 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:15:21.530727 9974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b4903da5afb4ba691b22a68307a3480 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:21.531814 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:15:21.532779 9974 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b4903da5afb4ba691b22a68307a3480 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b4903da5afb4ba691b22a68307a3480" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40101 } }
I20260926 15:15:21.533272 9974 leader_election.cc:302] T 00000000000000000000000000000000 P 2b4903da5afb4ba691b22a68307a3480 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b4903da5afb4ba691b22a68307a3480; no voters:
I20260926 15:15:21.533746 9974 leader_election.cc:288] T 00000000000000000000000000000000 P 2b4903da5afb4ba691b22a68307a3480 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:15:21.533860 9979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b4903da5afb4ba691b22a68307a3480 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:21.534466 9979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b4903da5afb4ba691b22a68307a3480 [term 1 LEADER]: Becoming Leader. State: Replica: 2b4903da5afb4ba691b22a68307a3480, State: Running, Role: LEADER
I20260926 15:15:21.535107 9979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b4903da5afb4ba691b22a68307a3480 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b4903da5afb4ba691b22a68307a3480" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40101 } }
I20260926 15:15:21.536923 9979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b4903da5afb4ba691b22a68307a3480 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b4903da5afb4ba691b22a68307a3480. Latest consensus state: current_term: 1 leader_uuid: "2b4903da5afb4ba691b22a68307a3480" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b4903da5afb4ba691b22a68307a3480" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40101 } } }
I20260926 15:15:21.537262 9979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b4903da5afb4ba691b22a68307a3480 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:21.537802 9981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b4903da5afb4ba691b22a68307a3480 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b4903da5afb4ba691b22a68307a3480" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b4903da5afb4ba691b22a68307a3480" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40101 } } }
I20260926 15:15:21.538125 9981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b4903da5afb4ba691b22a68307a3480 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:21.541638 9974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b4903da5afb4ba691b22a68307a3480 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:15:21.541812 9983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:15:21.549009 9983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:15:21.558290 9983 catalog_manager.cc:1347] Generated new cluster ID: ceede33565e84094b67f26795c6a20a5
I20260926 15:15:21.558400 9983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:15:21.568212 9983 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:15:21.569746 9983 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:15:21.577581 9983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b4903da5afb4ba691b22a68307a3480: Generated new TSK 0
I20260926 15:15:21.578425 9983 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:15:21.583865 9977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:21.572019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49718 (local address 127.6.209.254:40101)
0926 15:15:21.572198 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:21.572206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:21.576616 (+ 4410us) server_negotiation.cc:407] Connection header received
0926 15:15:21.576833 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:21.576839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:21.576910 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:21.577018 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:21.579064 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:21.579734 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:21.581120 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:21.581542 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:21.581675 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:21.582672 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:21.582676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:21.582679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:15:21.583164 (+ 485us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:21.583226 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:15:21.583268 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:21.583380 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:21.583497 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 15:15:21.583578 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:15:21.589593 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 9907
I20260926 15:15:21.590755 9969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:21.668314 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 9907
2026-09-26T15:15:21Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (5195 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:15:21 dist-test-slave-j6q7 krb5kdc[10008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:15:21 dist-test-slave-j6q7 krb5kdc[10008](info): set up 2 sockets
Sep 26 15:15:21 dist-test-slave-j6q7 krb5kdc[10008](info): commencing operation
krb5kdc: starting...
W20260926 15:15:22.664799 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.004s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:15:22 dist-test-slave-j6q7 krb5kdc[10008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:15:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:15:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:15:22.822266 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44247
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:36735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:44247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:23.096596 10021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:23.096891 10021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:23.097046 10021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:23.103484 10021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:15:23.103603 10021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:23.103696 10021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:23.103788 10021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:15:23.103866 10021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:15:23.111905 10021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:36735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:44247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:23.112965 10021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:23.114050 10021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:23.124279 10026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:23.124231 10029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:23.124306 10027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:23.125084 10021 server_base.cc:1034] running on GCE node
I20260926 15:15:23.125684 10021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:23.126786 10021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:23.127972 10021 hybrid_clock.cc:648] HybridClock initialized: now 1790435723127935 us; error 36 us; skew 500 ppm
I20260926 15:15:23.128259 10021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:23 dist-test-slave-j6q7 krb5kdc[10008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435723, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:23.130869 10021 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:15:23.132037 10021 webserver.cc:460] Webserver started at http://127.6.209.254:33033/ using document root <none> and password file <none>
I20260926 15:15:23.132572 10021 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:23.132690 10021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:23.132998 10021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:23.135506 10021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "d2bf840a9491462da330b9d18df04ab9"
format_stamp: "Formatted at 2026-09-26 15:15:23 on dist-test-slave-j6q7"
server_key: "67cabffc71b2cf0ec7214fea7ad9e2a2"
server_key_iv: "b421f2658e4c0eea13fc7fe6a37d5995"
server_key_version: "encryptionkey@0"
I20260926 15:15:23.136198 10021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "d2bf840a9491462da330b9d18df04ab9"
format_stamp: "Formatted at 2026-09-26 15:15:23 on dist-test-slave-j6q7"
server_key: "67cabffc71b2cf0ec7214fea7ad9e2a2"
server_key_iv: "b421f2658e4c0eea13fc7fe6a37d5995"
server_key_version: "encryptionkey@0"
I20260926 15:15:23.140372 10021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 15:15:23.143237 10036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:23.144251 10021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:15:23.144400 10021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "d2bf840a9491462da330b9d18df04ab9"
format_stamp: "Formatted at 2026-09-26 15:15:23 on dist-test-slave-j6q7"
server_key: "67cabffc71b2cf0ec7214fea7ad9e2a2"
server_key_iv: "b421f2658e4c0eea13fc7fe6a37d5995"
server_key_version: "encryptionkey@0"
I20260926 15:15:23.144554 10021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:23.162592 10021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:23.166159 10021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:23.166414 10021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:23.180631 10021 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:44247
I20260926 15:15:23.180649 10087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:44247 every 8 connection(s)
I20260926 15:15:23.181802 10021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:15:23.184546 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 10021
I20260926 15:15:23.184765 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:15:23.185012 10088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:23.185434 6983 external_mini_cluster.cc:1428] Setting key 4de095d65b98e524ed0b65c050f3c888
I20260926 15:15:23.197400 10088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2bf840a9491462da330b9d18df04ab9: Bootstrap starting.
Sep 26 15:15:23 dist-test-slave-j6q7 krb5kdc[10008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:23.200698 10088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2bf840a9491462da330b9d18df04ab9: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:23.202014 10088 log.cc:824] T 00000000000000000000000000000000 P d2bf840a9491462da330b9d18df04ab9: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:23.204937 10088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2bf840a9491462da330b9d18df04ab9: No bootstrap required, opened a new log
I20260926 15:15:23.206847 10091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:23.187235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60558 (local address 127.6.209.254:44247)
0926 15:15:23.187993 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:23.188006 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:23.188046 (+ 40us) server_negotiation.cc:407] Connection header received
0926 15:15:23.189499 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:23.189543 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:23.189962 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:23.190330 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:23.191325 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:23.193143 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:23.194269 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:23.195315 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:23.195490 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:23.199096 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:23.199105 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:23.199119 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:23.199165 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:23.202759 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:23.203440 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:23.203447 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:23.203454 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:23.203616 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:23.203939 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:23.203944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:23.203946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:23.204498 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:23.204765 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:23.205144 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 15:15:23.205539 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":316,"threads_started":1}
I20260926 15:15:23.212661 10088 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2bf840a9491462da330b9d18df04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2bf840a9491462da330b9d18df04ab9" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44247 } }
I20260926 15:15:23.213004 10088 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2bf840a9491462da330b9d18df04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:23.213074 10088 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2bf840a9491462da330b9d18df04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2bf840a9491462da330b9d18df04ab9, State: Initialized, Role: FOLLOWER
I20260926 15:15:23.213495 10088 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2bf840a9491462da330b9d18df04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2bf840a9491462da330b9d18df04ab9" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44247 } }
I20260926 15:15:23.213651 10088 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2bf840a9491462da330b9d18df04ab9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:15:23.213732 10088 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2bf840a9491462da330b9d18df04ab9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:15:23.213832 10088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2bf840a9491462da330b9d18df04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:23.215567 10088 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2bf840a9491462da330b9d18df04ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2bf840a9491462da330b9d18df04ab9" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44247 } }
I20260926 15:15:23.215940 10088 leader_election.cc:302] T 00000000000000000000000000000000 P d2bf840a9491462da330b9d18df04ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2bf840a9491462da330b9d18df04ab9; no voters:
I20260926 15:15:23.216401 10088 leader_election.cc:288] T 00000000000000000000000000000000 P d2bf840a9491462da330b9d18df04ab9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:15:23.216532 10093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2bf840a9491462da330b9d18df04ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:23.217119 10093 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2bf840a9491462da330b9d18df04ab9 [term 1 LEADER]: Becoming Leader. State: Replica: d2bf840a9491462da330b9d18df04ab9, State: Running, Role: LEADER
I20260926 15:15:23.217693 10093 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2bf840a9491462da330b9d18df04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2bf840a9491462da330b9d18df04ab9" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44247 } }
I20260926 15:15:23.218174 10088 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2bf840a9491462da330b9d18df04ab9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:15:23.219949 10095 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2bf840a9491462da330b9d18df04ab9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2bf840a9491462da330b9d18df04ab9. Latest consensus state: current_term: 1 leader_uuid: "d2bf840a9491462da330b9d18df04ab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2bf840a9491462da330b9d18df04ab9" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44247 } } }
I20260926 15:15:23.220224 10095 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2bf840a9491462da330b9d18df04ab9 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:23.220676 10094 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2bf840a9491462da330b9d18df04ab9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2bf840a9491462da330b9d18df04ab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2bf840a9491462da330b9d18df04ab9" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44247 } } }
I20260926 15:15:23.220974 10094 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2bf840a9491462da330b9d18df04ab9 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:23.221163 10100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:15:23.226260 10100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:15:23.233803 10100 catalog_manager.cc:1347] Generated new cluster ID: 90304c98e6d44b84b779783fad051504
I20260926 15:15:23.233911 10100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:15:23.243327 10100 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:15:23.244446 10100 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:15:23.253232 10100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2bf840a9491462da330b9d18df04ab9: Generated new TSK 0
I20260926 15:15:23.253865 10100 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
I20260926 15:15:23.298334 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44247
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:36735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:23.585574 10114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:23.585803 10114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:23.585889 10114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:23.592156 10114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:23.592237 10114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:23.592338 10114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:15:23.601089 10114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:36735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:23.602108 10114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:23.603194 10114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:23.614753 10122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:23.614794 10120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:23.614715 10119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:23.615434 10114 server_base.cc:1034] running on GCE node
I20260926 15:15:23.615904 10114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:23.616617 10114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:23.617812 10114 hybrid_clock.cc:648] HybridClock initialized: now 1790435723617758 us; error 71 us; skew 500 ppm
I20260926 15:15:23.618072 10114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:23 dist-test-slave-j6q7 krb5kdc[10008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435723, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:23.620879 10114 init.cc:377] Logged in from keytab as kudu/127.6.209.193@KRBTEST.COM (short username kudu)
I20260926 15:15:23.622114 10114 webserver.cc:460] Webserver started at http://127.6.209.193:39783/ using document root <none> and password file <none>
I20260926 15:15:23.622709 10114 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:23.622792 10114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:23.623023 10114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:23.625449 10114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "eb48b8ca8f0c438e9809bd8877c59b79"
format_stamp: "Formatted at 2026-09-26 15:15:23 on dist-test-slave-j6q7"
server_key: "0bc5cdaee95c0b930fc38cf8d2cd6067"
server_key_iv: "dac41fd348aa21d0bed9a7ae11ff87f2"
server_key_version: "encryptionkey@0"
I20260926 15:15:23.626112 10114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "eb48b8ca8f0c438e9809bd8877c59b79"
format_stamp: "Formatted at 2026-09-26 15:15:23 on dist-test-slave-j6q7"
server_key: "0bc5cdaee95c0b930fc38cf8d2cd6067"
server_key_iv: "dac41fd348aa21d0bed9a7ae11ff87f2"
server_key_version: "encryptionkey@0"
I20260926 15:15:23.630259 10114 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 15:15:23.633137 10129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:23.634276 10114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:15:23.634382 10114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "eb48b8ca8f0c438e9809bd8877c59b79"
format_stamp: "Formatted at 2026-09-26 15:15:23 on dist-test-slave-j6q7"
server_key: "0bc5cdaee95c0b930fc38cf8d2cd6067"
server_key_iv: "dac41fd348aa21d0bed9a7ae11ff87f2"
server_key_version: "encryptionkey@0"
I20260926 15:15:23.634531 10114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:23.647518 10114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:23.650676 10114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:23.650867 10114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:23.651656 10114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:23.652624 10114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:23.652690 10114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:23.652757 10114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:23.652796 10114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:23.684119 10114 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:43509
I20260926 15:15:23.684147 10241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:43509 every 8 connection(s)
I20260926 15:15:23.685338 10114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:15:23.692189 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 10114
I20260926 15:15:23.692404 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:15:23.692986 6983 external_mini_cluster.cc:1428] Setting key 21efe784c37621b925e9a6d2f8e74a4d
Sep 26 15:15:23 dist-test-slave-j6q7 krb5kdc[10008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435723, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:23.705633 10091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:23.687657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:35307 (local address 127.6.209.254:44247)
0926 15:15:23.687874 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:23.687881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:23.688892 (+ 1011us) server_negotiation.cc:407] Connection header received
0926 15:15:23.690574 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:23.690579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:23.690659 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:23.690762 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:23.692208 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:23.693106 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:23.694901 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:23.695369 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:23.695480 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:23.699210 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:23.699215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:23.699223 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:23.699265 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:23.702542 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:23.703266 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:23.703275 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:23.703281 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:23.703449 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:23.703861 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:23.703868 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:23.703874 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:23.704223 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:23.704359 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:23.704966 (+ 607us) server_negotiation.cc:299] Negotiation successful
0926 15:15:23.705208 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 15:15:23.706413 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:23.688092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44247 (local address 127.6.209.193:35307)
0926 15:15:23.688735 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:23.688766 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:23.690291 (+ 1525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:23.690974 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:23.690980 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:23.691415 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:23.692029 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:23.692052 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:23.693271 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:23.693276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:23.694758 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:23.694768 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:23.695590 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:23.695600 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:23.695916 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:23.696613 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:23.696642 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:23.699010 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:23.702686 (+ 3676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:23.702692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:23.702702 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:23.703106 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:23.703557 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:23.703560 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:23.703563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:23.703734 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:23.704336 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:23.704341 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:23.704591 (+ 250us) client_negotiation.cc:769] Sending connection context
0926 15:15:23.704885 (+ 294us) client_negotiation.cc:240] Negotiation successful
0926 15:15:23.705217 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":246,"threads_started":1}
I20260926 15:15:23.707846 10242 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44247
I20260926 15:15:23.708120 10242 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:23.708956 10242 heartbeater.cc:511] Master 127.6.209.254:44247 requested a full tablet report, sending...
I20260926 15:15:23.711043 10053 ts_manager.cc:194] Registered new tserver with Master: eb48b8ca8f0c438e9809bd8877c59b79 (127.6.209.193:43509)
WARNING: no policy specified for kudu/127.6.209.194@KRBTEST.COM; defaulting to no policy
I20260926 15:15:23.712532 10053 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.193@KRBTEST.COM'} at 127.6.209.193:35307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
I20260926 15:15:23.733341 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44247
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:36735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:24.017191 10247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:24.017406 10247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:24.017544 10247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:24.024086 10247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:24.024195 10247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:24.024355 10247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:15:24.032567 10247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:36735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:24.033649 10247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:24.034785 10247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:24.045848 10252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:24.045962 10253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:24.046372 10255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:24.046854 10247 server_base.cc:1034] running on GCE node
I20260926 15:15:24.047353 10247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:24.048123 10247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:24.049309 10247 hybrid_clock.cc:648] HybridClock initialized: now 1790435724049270 us; error 36 us; skew 500 ppm
I20260926 15:15:24.049559 10247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:24 dist-test-slave-j6q7 krb5kdc[10008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435724, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:24.052139 10247 init.cc:377] Logged in from keytab as kudu/127.6.209.194@KRBTEST.COM (short username kudu)
I20260926 15:15:24.053301 10247 webserver.cc:460] Webserver started at http://127.6.209.194:34251/ using document root <none> and password file <none>
I20260926 15:15:24.053830 10247 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:24.053956 10247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:24.054193 10247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:24.056897 10247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "c02616d3bb4b490d8c1f281c2e3de412"
format_stamp: "Formatted at 2026-09-26 15:15:24 on dist-test-slave-j6q7"
server_key: "ede0a0073195fec9d61f35dee56dd6e0"
server_key_iv: "204f7a92c91dc89e86477a56b46e1bb2"
server_key_version: "encryptionkey@0"
I20260926 15:15:24.057543 10247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "c02616d3bb4b490d8c1f281c2e3de412"
format_stamp: "Formatted at 2026-09-26 15:15:24 on dist-test-slave-j6q7"
server_key: "ede0a0073195fec9d61f35dee56dd6e0"
server_key_iv: "204f7a92c91dc89e86477a56b46e1bb2"
server_key_version: "encryptionkey@0"
I20260926 15:15:24.061630 10247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 15:15:24.064431 10262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:24.065567 10247 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:15:24.065727 10247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "c02616d3bb4b490d8c1f281c2e3de412"
format_stamp: "Formatted at 2026-09-26 15:15:24 on dist-test-slave-j6q7"
server_key: "ede0a0073195fec9d61f35dee56dd6e0"
server_key_iv: "204f7a92c91dc89e86477a56b46e1bb2"
server_key_version: "encryptionkey@0"
I20260926 15:15:24.065897 10247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:24.078874 10247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:24.081871 10247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:24.082122 10247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:24.082847 10247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:24.083887 10247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:24.083985 10247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:24.084115 10247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:24.084177 10247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:24.113848 10247 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:44757
I20260926 15:15:24.113921 10374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:44757 every 8 connection(s)
I20260926 15:15:24.114917 10247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:15:24.117377 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 10247
I20260926 15:15:24.117590 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:15:24.118034 6983 external_mini_cluster.cc:1428] Setting key c7ca8a2d1bbfd4e3fc351ff4cf47fcca
Sep 26 15:15:24 dist-test-slave-j6q7 krb5kdc[10008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435724, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:24.134236 10091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:24.117195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:40137 (local address 127.6.209.254:44247)
0926 15:15:24.117400 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:24.117408 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:24.118662 (+ 1254us) server_negotiation.cc:407] Connection header received
0926 15:15:24.120779 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:24.120784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:24.120879 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:24.120973 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:24.122876 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:24.123671 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:24.125850 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:24.126258 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:24.126381 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:24.129389 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:24.129393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:24.129396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:24.129422 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:24.131471 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:24.132029 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:24.132034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:24.132037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:24.132144 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:24.132538 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:24.132543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:24.132548 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:24.132886 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:24.133022 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:24.133848 (+ 826us) server_negotiation.cc:299] Negotiation successful
0926 15:15:24.133989 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
WARNING: no policy specified for kudu/127.6.209.195@KRBTEST.COM; defaulting to no policy
I20260926 15:15:24.135455 10377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:24.117667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44247 (local address 127.6.209.194:40137)
0926 15:15:24.118504 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:24.118551 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:24.120516 (+ 1965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:24.121244 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:24.121258 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:24.122006 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:24.122718 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:24.122757 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:24.123823 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:24.123830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:24.125711 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:24.125722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:24.126458 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:24.126465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:24.126685 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:24.127267 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:24.127304 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:24.129220 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:24.131582 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:24.131589 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:24.131601 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:24.131928 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:24.132234 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:24.132239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:24.132242 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:24.132397 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:24.133082 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:24.133090 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:24.133437 (+ 347us) client_negotiation.cc:769] Sending connection context
0926 15:15:24.133740 (+ 303us) client_negotiation.cc:240] Negotiation successful
0926 15:15:24.134262 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":312,"threads_started":1}
I20260926 15:15:24.136638 10375 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44247
I20260926 15:15:24.136946 10375 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.195@KRBTEST.COM" created.
I20260926 15:15:24.137778 10375 heartbeater.cc:511] Master 127.6.209.254:44247 requested a full tablet report, sending...
I20260926 15:15:24.139063 10053 ts_manager.cc:194] Registered new tserver with Master: c02616d3bb4b490d8c1f281c2e3de412 (127.6.209.194:44757)
I20260926 15:15:24.140183 10053 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.194@KRBTEST.COM'} at 127.6.209.194:40137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
I20260926 15:15:24.157866 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44247
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:36735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:24.450670 10380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:24.450883 10380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:24.450958 10380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:24.457154 10380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:24.457237 10380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:24.457331 10380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:15:24.465229 10380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:36735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:24.466192 10380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:24.467223 10380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:24.478590 10385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:24.478644 10386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:24.478675 10388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:24.479482 10380 server_base.cc:1034] running on GCE node
I20260926 15:15:24.479949 10380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:24.480669 10380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:24.481848 10380 hybrid_clock.cc:648] HybridClock initialized: now 1790435724481790 us; error 51 us; skew 500 ppm
I20260926 15:15:24.482076 10380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:24 dist-test-slave-j6q7 krb5kdc[10008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435724, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:24.484763 10380 init.cc:377] Logged in from keytab as kudu/127.6.209.195@KRBTEST.COM (short username kudu)
I20260926 15:15:24.486021 10380 webserver.cc:460] Webserver started at http://127.6.209.195:46223/ using document root <none> and password file <none>
I20260926 15:15:24.486557 10380 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:24.486641 10380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:24.486845 10380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:24.489283 10380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "2b91cb516d654a81a6f362fa50a4cf97"
format_stamp: "Formatted at 2026-09-26 15:15:24 on dist-test-slave-j6q7"
server_key: "82df6b221696dfda284ce3b5199ae86e"
server_key_iv: "2bf1d2d7484d5847a5fe8cfff1c44c8a"
server_key_version: "encryptionkey@0"
I20260926 15:15:24.489974 10380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "2b91cb516d654a81a6f362fa50a4cf97"
format_stamp: "Formatted at 2026-09-26 15:15:24 on dist-test-slave-j6q7"
server_key: "82df6b221696dfda284ce3b5199ae86e"
server_key_iv: "2bf1d2d7484d5847a5fe8cfff1c44c8a"
server_key_version: "encryptionkey@0"
I20260926 15:15:24.494244 10380 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:15:24.497145 10395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:24.498206 10380 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:15:24.498318 10380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "2b91cb516d654a81a6f362fa50a4cf97"
format_stamp: "Formatted at 2026-09-26 15:15:24 on dist-test-slave-j6q7"
server_key: "82df6b221696dfda284ce3b5199ae86e"
server_key_iv: "2bf1d2d7484d5847a5fe8cfff1c44c8a"
server_key_version: "encryptionkey@0"
I20260926 15:15:24.498492 10380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:24.511152 10380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:24.514410 10380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:24.514621 10380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:24.515384 10380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:24.516475 10380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:24.516539 10380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:24.516609 10380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:24.516651 10380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:24.545655 10380 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:37281
I20260926 15:15:24.545727 10507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:37281 every 8 connection(s)
I20260926 15:15:24.546765 10380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:15:24.550770 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 10380
I20260926 15:15:24.550957 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:15:24.551465 6983 external_mini_cluster.cc:1428] Setting key a8f541083cbcf5f00266c99f33b0c244
Sep 26 15:15:24 dist-test-slave-j6q7 krb5kdc[10008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435724, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:24.564124 10091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:24.549027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:43195 (local address 127.6.209.254:44247)
0926 15:15:24.549199 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:24.549204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:24.550349 (+ 1145us) server_negotiation.cc:407] Connection header received
0926 15:15:24.552629 (+ 2280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:24.552634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:24.552712 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:24.552786 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:24.554395 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:24.555034 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:24.556464 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:24.556875 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:24.556987 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:24.559925 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:24.559930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:24.559933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:24.559960 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:24.562070 (+ 2110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:24.562631 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:24.562634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:24.562636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:24.562726 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:24.563025 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:24.563028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:24.563030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:24.563233 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:24.563325 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:24.563799 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 15:15:24.563944 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 15:15:24.565213 10510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:24.549572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44247 (local address 127.6.209.195:43195)
0926 15:15:24.550204 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:24.550238 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:24.552380 (+ 2142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:24.553066 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:24.553076 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:24.553603 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:24.554265 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:24.554286 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:24.555154 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:24.555158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:24.556362 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:24.556371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:24.557064 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:24.557068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:24.557269 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:24.557865 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:24.557884 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:24.559737 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:24.562195 (+ 2458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:24.562200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:24.562209 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:24.562546 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:24.562804 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:24.562807 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:24.562809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:24.562948 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:24.563320 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:24.563324 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:24.563557 (+ 233us) client_negotiation.cc:769] Sending connection context
0926 15:15:24.563738 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 15:15:24.564128 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":257,"threads_started":1}
I20260926 15:15:24.566461 10508 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44247
I20260926 15:15:24.566737 10508 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:24.567350 10508 heartbeater.cc:511] Master 127.6.209.254:44247 requested a full tablet report, sending...
I20260926 15:15:24.568519 10053 ts_manager.cc:194] Registered new tserver with Master: 2b91cb516d654a81a6f362fa50a4cf97 (127.6.209.195:37281)
I20260926 15:15:24.569355 10053 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.195@KRBTEST.COM'} at 127.6.209.195:43195
I20260926 15:15:24.577912 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 15:15:24 dist-test-slave-j6q7 krb5kdc[10008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435724, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 15:15:24 dist-test-slave-j6q7 krb5kdc[10008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435724, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:24.618451 10091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:24.606999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60574 (local address 127.6.209.254:44247)
0926 15:15:24.607213 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:24.607221 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:24.607525 (+ 304us) server_negotiation.cc:407] Connection header received
0926 15:15:24.607644 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:24.607647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:24.607711 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:24.607785 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:24.608946 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:24.609536 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:24.610492 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:24.610947 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:24.611039 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:24.614157 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:24.614161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:24.614164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:24.614189 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:24.616297 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:24.616872 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:24.616875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:24.616877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:24.616992 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:24.617256 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:24.617258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:24.617260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:24.617499 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:24.617665 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:24.617971 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 15:15:24.618156 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 15:15:24.618954 10053 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60574
I20260926 15:15:24.620092 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 10114
Sep 26 15:15:24 dist-test-slave-j6q7 krb5kdc[10008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435724, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.209.193@KRBTEST.COM
I20260926 15:15:24.633455 10519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:24.620944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52048 (local address 127.6.209.193:43509)
0926 15:15:24.621570 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:24.621577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:24.621598 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:15:24.621646 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:24.621652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:24.621839 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:24.621999 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:24.623043 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:24.624014 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:24.625071 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:24.625588 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:24.625727 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:24.628940 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:24.628950 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:24.628961 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:24.628985 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:24.631073 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:24.631642 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:24.631648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:24.631650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:24.631751 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:24.632094 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:24.632099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:24.632102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:24.632372 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:24.632655 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:24.632888 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 15:15:24.633141 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":361,"threads_started":1}
W20260926 15:15:24.634526 10237 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52048
W20260926 15:15:24.635479 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 10114: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:15:24.635639 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 10114
I20260926 15:15:24.652153 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 10247
Sep 26 15:15:24 dist-test-slave-j6q7 krb5kdc[10008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435724, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.209.194@KRBTEST.COM
I20260926 15:15:24.664785 10520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:24.652735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.6.209.194:44757)
0926 15:15:24.653222 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:24.653228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:24.653250 (+ 22us) server_negotiation.cc:407] Connection header received
0926 15:15:24.653298 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:24.653304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:24.653475 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:24.653584 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:24.654523 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:24.655442 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:24.656601 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:24.657143 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:24.657233 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:24.660370 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:24.660378 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:24.660389 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:24.660415 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:24.662970 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:24.663410 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:24.663415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:24.663417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:24.663511 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:24.663758 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:24.663761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:24.663763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:24.664040 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:24.664237 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:24.664382 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 15:15:24.664540 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":262,"threads_started":1}
W20260926 15:15:24.665938 10370 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35290
W20260926 15:15:24.666802 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 10247: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:15:24.666946 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 10247
I20260926 15:15:24.680526 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 10380
Sep 26 15:15:24 dist-test-slave-j6q7 krb5kdc[10008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435724, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.209.195@KRBTEST.COM
I20260926 15:15:24.694342 10521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:24.681276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.6.209.195:37281)
0926 15:15:24.681800 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:24.681806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:24.681830 (+ 24us) server_negotiation.cc:407] Connection header received
0926 15:15:24.681879 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:24.681884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:24.682064 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:24.682173 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:24.683236 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:24.684213 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:24.685712 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:24.686213 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:24.686301 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:24.689881 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:24.689889 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:24.689898 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:24.689932 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:24.692207 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:24.692752 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:24.692756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:24.692758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:24.692908 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:24.693207 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:24.693210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:24.693212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:24.693486 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:24.693665 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:24.693856 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 15:15:24.694013 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":265,"threads_started":1}
W20260926 15:15:24.695410 10503 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33572
W20260926 15:15:24.696447 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 10380: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:15:24.696558 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 10380
I20260926 15:15:24.710592 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 10021
I20260926 15:15:24.711488 10083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:24.804078 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 10021
2026-09-26T15:15:24Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3143 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:15:24 dist-test-slave-j6q7 krb5kdc[10525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:15:24 dist-test-slave-j6q7 krb5kdc[10525](info): set up 2 sockets
Sep 26 15:15:24 dist-test-slave-j6q7 krb5kdc[10525](info): commencing operation
krb5kdc: starting...
W20260926 15:15:25.812510 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.006s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:15:25 dist-test-slave-j6q7 krb5kdc[10525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:15:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:15:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:15:25.972656 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:35819
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:35819
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:26.254344 10538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:26.254642 10538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:26.254798 10538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:26.261369 10538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:15:26.261494 10538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:26.261574 10538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:26.261655 10538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:15:26.261720 10538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:15:26.270032 10538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:35819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:35819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:26.271064 10538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:26.272150 10538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:26.281966 10543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:26.282003 10546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:26.281924 10544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:26.282778 10538 server_base.cc:1034] running on GCE node
I20260926 15:15:26.283377 10538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:26.284439 10538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:26.285619 10538 hybrid_clock.cc:648] HybridClock initialized: now 1790435726285559 us; error 54 us; skew 500 ppm
I20260926 15:15:26.285931 10538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:26 dist-test-slave-j6q7 krb5kdc[10525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435726, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:26.288574 10538 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:15:26.289840 10538 webserver.cc:460] Webserver started at http://127.6.209.254:43883/ using document root <none> and password file <none>
I20260926 15:15:26.290390 10538 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:26.290529 10538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:26.290807 10538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:26.293354 10538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "19509a5cfebb4c44bb6d5148f0974111"
format_stamp: "Formatted at 2026-09-26 15:15:26 on dist-test-slave-j6q7"
server_key: "c3278772926e91e42972a17e57d79779"
server_key_iv: "44c6ad89be1b52a049a892af4fdabb0a"
server_key_version: "encryptionkey@0"
I20260926 15:15:26.294031 10538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "19509a5cfebb4c44bb6d5148f0974111"
format_stamp: "Formatted at 2026-09-26 15:15:26 on dist-test-slave-j6q7"
server_key: "c3278772926e91e42972a17e57d79779"
server_key_iv: "44c6ad89be1b52a049a892af4fdabb0a"
server_key_version: "encryptionkey@0"
I20260926 15:15:26.298348 10538 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:15:26.301147 10553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:26.302294 10538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:15:26.302457 10538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "19509a5cfebb4c44bb6d5148f0974111"
format_stamp: "Formatted at 2026-09-26 15:15:26 on dist-test-slave-j6q7"
server_key: "c3278772926e91e42972a17e57d79779"
server_key_iv: "44c6ad89be1b52a049a892af4fdabb0a"
server_key_version: "encryptionkey@0"
I20260926 15:15:26.302636 10538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:26.320005 10538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:26.323354 10538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:26.323606 10538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:26.339550 10538 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:35819
I20260926 15:15:26.339582 10604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:35819 every 8 connection(s)
I20260926 15:15:26.340580 10538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:15:26.343827 10605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:26.346977 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 10538
I20260926 15:15:26.347146 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:15:26.347765 6983 external_mini_cluster.cc:1428] Setting key e90dad58b844bbce03588b547dfdbd53
I20260926 15:15:26.355204 10605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19509a5cfebb4c44bb6d5148f0974111: Bootstrap starting.
I20260926 15:15:26.358796 10605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19509a5cfebb4c44bb6d5148f0974111: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:26.360041 10605 log.cc:824] T 00000000000000000000000000000000 P 19509a5cfebb4c44bb6d5148f0974111: Log is configured to *not* fsync() on all Append() calls
Sep 26 15:15:26 dist-test-slave-j6q7 krb5kdc[10525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:26.362993 10605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19509a5cfebb4c44bb6d5148f0974111: No bootstrap required, opened a new log
I20260926 15:15:26.367754 10608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:26.349506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60338 (local address 127.6.209.254:35819)
0926 15:15:26.350111 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:26.350120 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:26.350152 (+ 32us) server_negotiation.cc:407] Connection header received
0926 15:15:26.351547 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:26.351564 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:26.351870 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:26.352133 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:26.353734 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:26.354841 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:26.356665 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:26.357279 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:26.357381 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:26.361751 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:26.361758 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:26.361770 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:26.361799 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:26.364255 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:26.364870 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:26.364876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:26.364880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:26.365015 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:26.365446 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:26.365451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:26.365453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:26.365897 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:26.366181 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:26.366616 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 15:15:26.366926 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":295,"threads_started":1}
I20260926 15:15:26.369890 10605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19509a5cfebb4c44bb6d5148f0974111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19509a5cfebb4c44bb6d5148f0974111" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35819 } }
I20260926 15:15:26.370245 10605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19509a5cfebb4c44bb6d5148f0974111 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:26.370355 10605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19509a5cfebb4c44bb6d5148f0974111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19509a5cfebb4c44bb6d5148f0974111, State: Initialized, Role: FOLLOWER
I20260926 15:15:26.370853 10605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19509a5cfebb4c44bb6d5148f0974111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19509a5cfebb4c44bb6d5148f0974111" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35819 } }
I20260926 15:15:26.371049 10605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19509a5cfebb4c44bb6d5148f0974111 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:15:26.371155 10605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19509a5cfebb4c44bb6d5148f0974111 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:15:26.371308 10605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19509a5cfebb4c44bb6d5148f0974111 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:26.373210 10605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19509a5cfebb4c44bb6d5148f0974111 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19509a5cfebb4c44bb6d5148f0974111" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35819 } }
I20260926 15:15:26.373591 10605 leader_election.cc:302] T 00000000000000000000000000000000 P 19509a5cfebb4c44bb6d5148f0974111 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19509a5cfebb4c44bb6d5148f0974111; no voters:
I20260926 15:15:26.374049 10605 leader_election.cc:288] T 00000000000000000000000000000000 P 19509a5cfebb4c44bb6d5148f0974111 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:15:26.374161 10610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19509a5cfebb4c44bb6d5148f0974111 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:26.374752 10610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19509a5cfebb4c44bb6d5148f0974111 [term 1 LEADER]: Becoming Leader. State: Replica: 19509a5cfebb4c44bb6d5148f0974111, State: Running, Role: LEADER
I20260926 15:15:26.375202 10610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19509a5cfebb4c44bb6d5148f0974111 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19509a5cfebb4c44bb6d5148f0974111" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35819 } }
I20260926 15:15:26.375676 10605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19509a5cfebb4c44bb6d5148f0974111 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:15:26.377561 10611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19509a5cfebb4c44bb6d5148f0974111 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19509a5cfebb4c44bb6d5148f0974111" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19509a5cfebb4c44bb6d5148f0974111" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35819 } } }
I20260926 15:15:26.377908 10611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19509a5cfebb4c44bb6d5148f0974111 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:26.377931 10612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19509a5cfebb4c44bb6d5148f0974111 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19509a5cfebb4c44bb6d5148f0974111. Latest consensus state: current_term: 1 leader_uuid: "19509a5cfebb4c44bb6d5148f0974111" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19509a5cfebb4c44bb6d5148f0974111" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35819 } } }
I20260926 15:15:26.378376 10612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19509a5cfebb4c44bb6d5148f0974111 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:26.381845 10619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:15:26.386677 10619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:15:26.394222 10619 catalog_manager.cc:1347] Generated new cluster ID: 74b52efdac1541b3b0f807a8cb42fd2a
I20260926 15:15:26.394332 10619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:15:26.403908 10619 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:15:26.405396 10619 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:15:26.414800 10619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19509a5cfebb4c44bb6d5148f0974111: Generated new TSK 0
I20260926 15:15:26.415445 10619 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
I20260926 15:15:26.467898 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35819
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:26.775911 10631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:26.776222 10631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:26.776391 10631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:26.782796 10631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:26.782871 10631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:26.783006 10631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:15:26.790958 10631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:26.792002 10631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:26.793054 10631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:26.805160 10639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:26.805208 10636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:26.805199 10637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:26.806030 10631 server_base.cc:1034] running on GCE node
I20260926 15:15:26.806649 10631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:26.807459 10631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:26.808670 10631 hybrid_clock.cc:648] HybridClock initialized: now 1790435726808636 us; error 50 us; skew 500 ppm
I20260926 15:15:26.808964 10631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:26 dist-test-slave-j6q7 krb5kdc[10525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435726, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:26.818748 10631 init.cc:377] Logged in from keytab as kudu/127.6.209.193@KRBTEST.COM (short username kudu)
I20260926 15:15:26.819854 10631 webserver.cc:460] Webserver started at http://127.6.209.193:42619/ using document root <none> and password file <none>
I20260926 15:15:26.820365 10631 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:26.820454 10631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:26.820659 10631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:26.823195 10631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "2c0b25db46f24f7d8c09e3bc372fd5d7"
format_stamp: "Formatted at 2026-09-26 15:15:26 on dist-test-slave-j6q7"
server_key: "3f284946d0a4fbfdacb005331572e160"
server_key_iv: "5ba6d0c01f50a29bfceb624d17f49e3f"
server_key_version: "encryptionkey@0"
I20260926 15:15:26.823814 10631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "2c0b25db46f24f7d8c09e3bc372fd5d7"
format_stamp: "Formatted at 2026-09-26 15:15:26 on dist-test-slave-j6q7"
server_key: "3f284946d0a4fbfdacb005331572e160"
server_key_iv: "5ba6d0c01f50a29bfceb624d17f49e3f"
server_key_version: "encryptionkey@0"
I20260926 15:15:26.828094 10631 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 15:15:26.830971 10646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:26.832018 10631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:15:26.832170 10631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "2c0b25db46f24f7d8c09e3bc372fd5d7"
format_stamp: "Formatted at 2026-09-26 15:15:26 on dist-test-slave-j6q7"
server_key: "3f284946d0a4fbfdacb005331572e160"
server_key_iv: "5ba6d0c01f50a29bfceb624d17f49e3f"
server_key_version: "encryptionkey@0"
I20260926 15:15:26.832325 10631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:26.849535 10631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:26.853034 10631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:26.853281 10631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:26.854043 10631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:26.855144 10631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:26.855216 10631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:26.855363 10631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:26.855425 10631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:26.886806 10631 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:36717
I20260926 15:15:26.886878 10758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:36717 every 8 connection(s)
I20260926 15:15:26.887878 10631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:15:26.892930 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 10631
I20260926 15:15:26.893105 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:15:26.893661 6983 external_mini_cluster.cc:1428] Setting key 1502636cfa8ed1d7869a2f193f58cb4a
Sep 26 15:15:26 dist-test-slave-j6q7 krb5kdc[10525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435726, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:26.905207 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:26.889988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:51319 (local address 127.6.209.254:35819)
0926 15:15:26.890334 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:26.890337 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:26.891196 (+ 859us) server_negotiation.cc:407] Connection header received
0926 15:15:26.892746 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:26.892750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:26.892871 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:26.892978 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:26.894293 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:26.895022 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:26.896522 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:26.896935 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:26.897054 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:26.899803 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:26.899814 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:26.899817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:26.899841 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:26.902178 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:26.902982 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:26.902985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:26.902987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:26.903091 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:26.903545 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:26.903548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:26.903550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:26.903782 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:26.903875 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:26.904698 (+ 823us) server_negotiation.cc:299] Negotiation successful
0926 15:15:26.904879 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":181,"threads_started":1}
I20260926 15:15:26.906497 10762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:26.890446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:35819 (local address 127.6.209.193:51319)
0926 15:15:26.891051 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:26.891086 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:26.892520 (+ 1434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:26.893202 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:26.893212 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:26.893648 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:26.894153 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:26.894166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:26.895194 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:26.895200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:26.896394 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:26.896406 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:26.897131 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:26.897141 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:26.897332 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:26.897888 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:26.897919 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:26.899636 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:26.902317 (+ 2681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:26.902326 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:26.902340 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:26.902873 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:26.903189 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:26.903196 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:26.903201 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:26.903444 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:26.903925 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:26.903936 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:26.904343 (+ 407us) client_negotiation.cc:769] Sending connection context
0926 15:15:26.904642 (+ 299us) client_negotiation.cc:240] Negotiation successful
0926 15:15:26.905159 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":217,"threads_started":1}
I20260926 15:15:26.907879 10759 heartbeater.cc:347] Connected to a master server at 127.6.209.254:35819
I20260926 15:15:26.908172 10759 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:26.908746 10759 heartbeater.cc:511] Master 127.6.209.254:35819 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.209.194@KRBTEST.COM; defaulting to no policy
I20260926 15:15:26.910589 10570 ts_manager.cc:194] Registered new tserver with Master: 2c0b25db46f24f7d8c09e3bc372fd5d7 (127.6.209.193:36717)
I20260926 15:15:26.911926 10570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.193@KRBTEST.COM'} at 127.6.209.193:51319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
I20260926 15:15:26.933621 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35819
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:27.225036 10765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:27.225348 10765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:27.225538 10765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:27.232089 10765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:27.232230 10765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:27.232396 10765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:15:27.241430 10765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:27.242621 10765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:27.243858 10765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:27.255153 10770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:27.255204 10771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:27.255112 10773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:27.256138 10765 server_base.cc:1034] running on GCE node
I20260926 15:15:27.256719 10765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:27.257576 10765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:27.258775 10765 hybrid_clock.cc:648] HybridClock initialized: now 1790435727258737 us; error 50 us; skew 500 ppm
I20260926 15:15:27.259051 10765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:27 dist-test-slave-j6q7 krb5kdc[10525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435727, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:27.261808 10765 init.cc:377] Logged in from keytab as kudu/127.6.209.194@KRBTEST.COM (short username kudu)
I20260926 15:15:27.262954 10765 webserver.cc:460] Webserver started at http://127.6.209.194:40449/ using document root <none> and password file <none>
I20260926 15:15:27.263469 10765 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:27.263574 10765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:27.263789 10765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:27.266355 10765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "cd3ecdffcd2e4f1699cc21e3bd401ba6"
format_stamp: "Formatted at 2026-09-26 15:15:27 on dist-test-slave-j6q7"
server_key: "771c78a400b0244fbb6471a361fb072c"
server_key_iv: "bdd70197cf4b8dde81026a9a5b3954d8"
server_key_version: "encryptionkey@0"
I20260926 15:15:27.266965 10765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "cd3ecdffcd2e4f1699cc21e3bd401ba6"
format_stamp: "Formatted at 2026-09-26 15:15:27 on dist-test-slave-j6q7"
server_key: "771c78a400b0244fbb6471a361fb072c"
server_key_iv: "bdd70197cf4b8dde81026a9a5b3954d8"
server_key_version: "encryptionkey@0"
I20260926 15:15:27.271329 10765 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 15:15:27.274114 10780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:27.275137 10765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 15:15:27.275281 10765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "cd3ecdffcd2e4f1699cc21e3bd401ba6"
format_stamp: "Formatted at 2026-09-26 15:15:27 on dist-test-slave-j6q7"
server_key: "771c78a400b0244fbb6471a361fb072c"
server_key_iv: "bdd70197cf4b8dde81026a9a5b3954d8"
server_key_version: "encryptionkey@0"
I20260926 15:15:27.275422 10765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:27.294642 10765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:27.297945 10765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:27.298156 10765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:27.298835 10765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:27.299825 10765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:27.299898 10765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:27.299968 10765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:27.300005 10765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:27.329530 10765 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:37035
I20260926 15:15:27.329615 10892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:37035 every 8 connection(s)
I20260926 15:15:27.330737 10765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:15:27.338318 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 10765
I20260926 15:15:27.338478 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:15:27.339015 6983 external_mini_cluster.cc:1428] Setting key 5d36528e2a9a0e65914e5b894bd12d06
Sep 26 15:15:27 dist-test-slave-j6q7 krb5kdc[10525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435727, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:27.349493 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:27.332916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:56189 (local address 127.6.209.254:35819)
0926 15:15:27.333056 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:27.333060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:27.334017 (+ 957us) server_negotiation.cc:407] Connection header received
0926 15:15:27.335656 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:27.335660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:27.335725 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:27.335810 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:27.337296 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:27.337930 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:27.339853 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:27.340424 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:27.340538 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:27.343625 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:27.343629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:27.343632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:27.343658 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:27.346359 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:27.347057 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:27.347065 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:27.347070 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:27.347228 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:27.347691 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:27.347696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:27.347701 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:27.348042 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:27.348213 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:27.348894 (+ 681us) server_negotiation.cc:299] Negotiation successful
0926 15:15:27.349128 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 15:15:27.350209 10895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:27.333305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:35819 (local address 127.6.209.194:56189)
0926 15:15:27.333882 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:27.333913 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:27.335430 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:27.336028 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:27.336040 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:27.336592 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:27.337163 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:27.337182 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:27.338081 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:27.338088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:27.339714 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:27.339728 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:27.340614 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:27.340622 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:27.340900 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:27.341551 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:27.341571 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:27.343429 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:27.346498 (+ 3069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:27.346506 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:27.346518 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:27.346936 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:27.347355 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:27.347359 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:27.347363 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:27.347570 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:27.348167 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:27.348174 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:27.348486 (+ 312us) client_negotiation.cc:769] Sending connection context
0926 15:15:27.348728 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 15:15:27.349168 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":217,"threads_started":1}
I20260926 15:15:27.351374 10893 heartbeater.cc:347] Connected to a master server at 127.6.209.254:35819
I20260926 15:15:27.351665 10893 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:27.352375 10893 heartbeater.cc:511] Master 127.6.209.254:35819 requested a full tablet report, sending...
I20260926 15:15:27.353801 10570 ts_manager.cc:194] Registered new tserver with Master: cd3ecdffcd2e4f1699cc21e3bd401ba6 (127.6.209.194:37035)
I20260926 15:15:27.354803 10570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.194@KRBTEST.COM'} at 127.6.209.194:56189
WARNING: no policy specified for kudu/127.6.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
I20260926 15:15:27.379714 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35819
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:27.666904 10898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:27.667169 10898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:27.667248 10898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:27.673529 10898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:27.673635 10898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:27.673796 10898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:15:27.682626 10898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:27.683696 10898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:27.684831 10898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:27.696224 10903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:27.696182 10906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:27.696267 10904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:27.697043 10898 server_base.cc:1034] running on GCE node
I20260926 15:15:27.697587 10898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:27.698343 10898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:27.699545 10898 hybrid_clock.cc:648] HybridClock initialized: now 1790435727699513 us; error 44 us; skew 500 ppm
I20260926 15:15:27.699797 10898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:27 dist-test-slave-j6q7 krb5kdc[10525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435727, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:27.702575 10898 init.cc:377] Logged in from keytab as kudu/127.6.209.195@KRBTEST.COM (short username kudu)
I20260926 15:15:27.703701 10898 webserver.cc:460] Webserver started at http://127.6.209.195:33233/ using document root <none> and password file <none>
I20260926 15:15:27.704295 10898 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:27.704424 10898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:27.704691 10898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:27.707279 10898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "0ad90cf15bbf4fd2a5a02ea85e4b12f5"
format_stamp: "Formatted at 2026-09-26 15:15:27 on dist-test-slave-j6q7"
server_key: "69fb43317e0d5e4ed22ac83dff837ea7"
server_key_iv: "788a4f1fb3827b66fd5dee31543d24e5"
server_key_version: "encryptionkey@0"
I20260926 15:15:27.707921 10898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "0ad90cf15bbf4fd2a5a02ea85e4b12f5"
format_stamp: "Formatted at 2026-09-26 15:15:27 on dist-test-slave-j6q7"
server_key: "69fb43317e0d5e4ed22ac83dff837ea7"
server_key_iv: "788a4f1fb3827b66fd5dee31543d24e5"
server_key_version: "encryptionkey@0"
I20260926 15:15:27.711987 10898 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 15:15:27.714823 10913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:27.715922 10898 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:15:27.716085 10898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "0ad90cf15bbf4fd2a5a02ea85e4b12f5"
format_stamp: "Formatted at 2026-09-26 15:15:27 on dist-test-slave-j6q7"
server_key: "69fb43317e0d5e4ed22ac83dff837ea7"
server_key_iv: "788a4f1fb3827b66fd5dee31543d24e5"
server_key_version: "encryptionkey@0"
I20260926 15:15:27.716276 10898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:27.732115 10898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:27.735409 10898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:27.735654 10898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:27.736516 10898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:27.737617 10898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:27.737710 10898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:27.737856 10898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:27.737926 10898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:27.766371 10898 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:34995
I20260926 15:15:27.766462 11025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:34995 every 8 connection(s)
I20260926 15:15:27.767376 10898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:15:27.773427 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 10898
I20260926 15:15:27.773576 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:15:27.773984 6983 external_mini_cluster.cc:1428] Setting key 43d1691b54277464f800e217d5a9548d
Sep 26 15:15:27 dist-test-slave-j6q7 krb5kdc[10525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435727, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:27.784336 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:27.769419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:47631 (local address 127.6.209.254:35819)
0926 15:15:27.769585 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:27.769589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:27.770543 (+ 954us) server_negotiation.cc:407] Connection header received
0926 15:15:27.772067 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:27.772071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:27.772132 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:27.772213 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:27.773645 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:27.774376 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:27.776203 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:27.776670 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:27.776749 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:27.779831 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:27.779835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:27.779838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:27.779865 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:27.782218 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:27.782787 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:27.782790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:27.782792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:27.782886 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:27.783174 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:27.783177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:27.783178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:27.783416 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:27.783534 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:27.784008 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 15:15:27.784131 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 15:15:27.785172 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:27.769746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:35819 (local address 127.6.209.195:47631)
0926 15:15:27.770392 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:27.770425 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:27.771811 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:27.772379 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:27.772386 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:27.772797 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:27.773421 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:27.773443 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:27.774517 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:27.774520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:27.776061 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:27.776074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:27.776889 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:27.776896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:27.777098 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:27.777652 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:27.777675 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:27.779665 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:27.782323 (+ 2658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:27.782329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:27.782337 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:27.782706 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:27.782964 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:27.782966 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:27.782968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:27.783098 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:27.783518 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:27.783522 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:27.783750 (+ 228us) client_negotiation.cc:769] Sending connection context
0926 15:15:27.783938 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 15:15:27.784211 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":274,"threads_started":1}
I20260926 15:15:27.786370 11026 heartbeater.cc:347] Connected to a master server at 127.6.209.254:35819
I20260926 15:15:27.786602 11026 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:27.787179 11026 heartbeater.cc:511] Master 127.6.209.254:35819 requested a full tablet report, sending...
I20260926 15:15:27.788596 10570 ts_manager.cc:194] Registered new tserver with Master: 0ad90cf15bbf4fd2a5a02ea85e4b12f5 (127.6.209.195:34995)
I20260926 15:15:27.789264 10570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.195@KRBTEST.COM'} at 127.6.209.195:47631
I20260926 15:15:27.800002 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 15:15:27 dist-test-slave-j6q7 krb5kdc[10525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 15:15:27 dist-test-slave-j6q7 krb5kdc[10525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.193@KRBTEST.COM
I20260926 15:15:27.841784 11033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:27.828109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37972 (local address 127.6.209.193:36717)
0926 15:15:27.828600 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:27.828606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:27.828626 (+ 20us) server_negotiation.cc:407] Connection header received
0926 15:15:27.828675 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:27.828680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:27.828923 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:27.829079 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:27.830340 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:27.831334 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:27.832476 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:27.833069 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:27.833204 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:27.836660 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:27.836669 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:27.836682 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:27.836713 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:27.839159 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:27.839834 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:27.839842 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:27.839845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:27.839975 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:27.840320 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:27.840326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:27.840329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:27.840624 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:27.840900 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:27.841220 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 15:15:27.841474 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":220,"threads_started":1}
Sep 26 15:15:27 dist-test-slave-j6q7 krb5kdc[10525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:27.856691 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:27.845518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60348 (local address 127.6.209.254:35819)
0926 15:15:27.845712 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:27.845718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:27.845888 (+ 170us) server_negotiation.cc:407] Connection header received
0926 15:15:27.846069 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:27.846076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:27.846139 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:27.846239 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:27.847352 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:27.847985 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:27.849084 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:27.849424 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:27.849534 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:27.852422 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:27.852426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:27.852429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:27.852453 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:27.854414 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:27.854920 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:27.854924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:27.854928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:27.855034 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:27.855372 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:27.855378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:27.855382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:27.855661 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:27.855898 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:27.856198 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 15:15:27.856426 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 15:15:27.857187 10570 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60348
I20260926 15:15:27.858011 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 10631
I20260926 15:15:27.869304 11033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:27.858712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37976 (local address 127.6.209.193:36717)
0926 15:15:27.858870 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:27.858874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:27.858975 (+ 101us) server_negotiation.cc:407] Connection header received
0926 15:15:27.859107 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:27.859110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:27.859164 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:27.859235 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:27.860230 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:27.860795 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:27.862473 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:27.863005 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:27.863106 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:27.865354 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:27.865358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:27.865361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:27.865387 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:27.867453 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:27.867927 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:27.867930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:27.867933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:27.868022 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:27.868296 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:27.868299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:27.868300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:27.868499 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:27.868636 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:27.868855 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 15:15:27.868998 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 15:15:27.869825 10754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:27.942525 10759 heartbeater.cc:503] Master 127.6.209.254:35819 was elected leader, sending a full tablet report...
I20260926 15:15:27.942975 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 10631
W20260926 15:15:27.957803 10556 connection.cc:535] server connection from 127.6.209.193:51319 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 15:15:27.958019 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 10765
Sep 26 15:15:27 dist-test-slave-j6q7 krb5kdc[10525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.194@KRBTEST.COM
I20260926 15:15:27.972710 11038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:27.958767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47194 (local address 127.6.209.194:37035)
0926 15:15:27.959222 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:27.959228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:27.959251 (+ 23us) server_negotiation.cc:407] Connection header received
0926 15:15:27.959300 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:27.959305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:27.959506 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:27.959659 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:27.960731 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:27.961919 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:27.963231 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:27.963719 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:27.963841 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:27.967625 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:27.967640 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:27.967652 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:27.967701 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:27.970159 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:27.970795 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:27.970803 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:27.970805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:27.970918 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:27.971259 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:27.971266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:27.971269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:27.971574 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:27.971869 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:27.972163 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 15:15:27.972422 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":225,"threads_started":1}
I20260926 15:15:27.973814 10888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:28.049141 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 10765
I20260926 15:15:28.063805 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 10898
Sep 26 15:15:28 dist-test-slave-j6q7 krb5kdc[10525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.195@KRBTEST.COM
I20260926 15:15:28.078258 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:28.064640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34970 (local address 127.6.209.195:34995)
0926 15:15:28.065228 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:28.065234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:28.065260 (+ 26us) server_negotiation.cc:407] Connection header received
0926 15:15:28.065337 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:28.065343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:28.065550 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:28.065719 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:28.066941 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:28.067871 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:28.069020 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:28.069487 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:28.069609 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:28.073344 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:28.073354 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:28.073366 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:28.073401 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:28.075770 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:28.076377 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:28.076394 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:28.076396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:28.076584 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:28.076989 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:28.076995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:28.076998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:28.077283 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:28.077556 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:28.077758 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 15:15:28.077980 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":320,"threads_started":1}
I20260926 15:15:28.079419 11021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:28.155485 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 10898
I20260926 15:15:28.170862 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 10538
I20260926 15:15:28.171855 10600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:28.261202 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 10538
2026-09-26T15:15:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3453 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:15:28 dist-test-slave-j6q7 krb5kdc[11045](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:15:28 dist-test-slave-j6q7 krb5kdc[11045](info): set up 2 sockets
Sep 26 15:15:28 dist-test-slave-j6q7 krb5kdc[11045](info): commencing operation
krb5kdc: starting...
W20260926 15:15:29.249100 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:15:29 dist-test-slave-j6q7 krb5kdc[11045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:15:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:15:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:15:29.411288 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:38555
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:38555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:29.689781 11058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:29.690038 11058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:29.690183 11058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:29.696208 11058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:15:29.696337 11058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:29.696446 11058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:29.696548 11058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:15:29.696645 11058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:15:29.705098 11058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:38555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:38555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:29.706176 11058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:29.707324 11058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:29.717233 11066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:29.717298 11064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:29.717350 11063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:29.718094 11058 server_base.cc:1034] running on GCE node
I20260926 15:15:29.718672 11058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:29.719758 11058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:29.720942 11058 hybrid_clock.cc:648] HybridClock initialized: now 1790435729720902 us; error 39 us; skew 500 ppm
I20260926 15:15:29.721273 11058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:29 dist-test-slave-j6q7 krb5kdc[11045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435729, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:29.723994 11058 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:15:29.724939 11058 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:29.725014 11058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:29.725250 11058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:29.727764 11058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "ee79f57554cb4d089a9cfb85070567ce"
format_stamp: "Formatted at 2026-09-26 15:15:29 on dist-test-slave-j6q7"
server_key: "16352d332ac58cc61176bffaf5efd92e"
server_key_iv: "9fcede072bfb54d138f3f3a3f40efc98"
server_key_version: "encryptionkey@0"
I20260926 15:15:29.728395 11058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "ee79f57554cb4d089a9cfb85070567ce"
format_stamp: "Formatted at 2026-09-26 15:15:29 on dist-test-slave-j6q7"
server_key: "16352d332ac58cc61176bffaf5efd92e"
server_key_iv: "9fcede072bfb54d138f3f3a3f40efc98"
server_key_version: "encryptionkey@0"
I20260926 15:15:29.732538 11058 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260926 15:15:29.735345 11072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:29.736498 11058 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:15:29.736675 11058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "ee79f57554cb4d089a9cfb85070567ce"
format_stamp: "Formatted at 2026-09-26 15:15:29 on dist-test-slave-j6q7"
server_key: "16352d332ac58cc61176bffaf5efd92e"
server_key_iv: "9fcede072bfb54d138f3f3a3f40efc98"
server_key_version: "encryptionkey@0"
I20260926 15:15:29.736891 11058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:29.747685 11058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:29.751434 11058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:29.751675 11058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:29.768036 11058 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:38555
I20260926 15:15:29.768054 11123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:38555 every 8 connection(s)
I20260926 15:15:29.768738 11058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:15:29.771589 11124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:29.774345 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 11058
I20260926 15:15:29.774554 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:15:29.775318 6983 external_mini_cluster.cc:1428] Setting key 3c1f071900efa6ec3b5c95d0dfc5f304
I20260926 15:15:29.783219 11124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee79f57554cb4d089a9cfb85070567ce: Bootstrap starting.
I20260926 15:15:29.786891 11124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee79f57554cb4d089a9cfb85070567ce: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:29.788038 11124 log.cc:824] T 00000000000000000000000000000000 P ee79f57554cb4d089a9cfb85070567ce: Log is configured to *not* fsync() on all Append() calls
Sep 26 15:15:29 dist-test-slave-j6q7 krb5kdc[11045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:29.790462 11124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee79f57554cb4d089a9cfb85070567ce: No bootstrap required, opened a new log
I20260926 15:15:29.796495 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:29.777198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40946 (local address 127.6.209.254:38555)
0926 15:15:29.777912 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:29.777922 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:29.777955 (+ 33us) server_negotiation.cc:407] Connection header received
0926 15:15:29.779352 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:29.779378 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:29.779719 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:29.779996 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:29.781703 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:29.782858 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:29.784565 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:29.785633 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:29.785759 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:29.790017 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:29.790023 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:29.790034 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:29.790062 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:29.792767 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:29.793434 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:29.793440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:29.793447 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:29.793590 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:29.793932 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:29.793935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:29.793937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:29.794382 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:29.794671 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:29.795010 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 15:15:29.795338 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":350,"threads_started":1}
I20260926 15:15:29.797611 11124 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee79f57554cb4d089a9cfb85070567ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee79f57554cb4d089a9cfb85070567ce" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38555 } }
I20260926 15:15:29.797909 11124 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee79f57554cb4d089a9cfb85070567ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:29.797996 11124 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee79f57554cb4d089a9cfb85070567ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee79f57554cb4d089a9cfb85070567ce, State: Initialized, Role: FOLLOWER
I20260926 15:15:29.798532 11124 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee79f57554cb4d089a9cfb85070567ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee79f57554cb4d089a9cfb85070567ce" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38555 } }
I20260926 15:15:29.798731 11124 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee79f57554cb4d089a9cfb85070567ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:15:29.798884 11124 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee79f57554cb4d089a9cfb85070567ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:15:29.799039 11124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee79f57554cb4d089a9cfb85070567ce [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:29.800774 11124 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee79f57554cb4d089a9cfb85070567ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee79f57554cb4d089a9cfb85070567ce" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38555 } }
I20260926 15:15:29.801245 11124 leader_election.cc:302] T 00000000000000000000000000000000 P ee79f57554cb4d089a9cfb85070567ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee79f57554cb4d089a9cfb85070567ce; no voters:
I20260926 15:15:29.801715 11124 leader_election.cc:288] T 00000000000000000000000000000000 P ee79f57554cb4d089a9cfb85070567ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:15:29.801819 11129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee79f57554cb4d089a9cfb85070567ce [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:29.802484 11129 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee79f57554cb4d089a9cfb85070567ce [term 1 LEADER]: Becoming Leader. State: Replica: ee79f57554cb4d089a9cfb85070567ce, State: Running, Role: LEADER
I20260926 15:15:29.802923 11129 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee79f57554cb4d089a9cfb85070567ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee79f57554cb4d089a9cfb85070567ce" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38555 } }
I20260926 15:15:29.803514 11124 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee79f57554cb4d089a9cfb85070567ce [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:15:29.805094 11130 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee79f57554cb4d089a9cfb85070567ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee79f57554cb4d089a9cfb85070567ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee79f57554cb4d089a9cfb85070567ce" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38555 } } }
I20260926 15:15:29.805410 11130 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee79f57554cb4d089a9cfb85070567ce [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:29.806109 11131 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee79f57554cb4d089a9cfb85070567ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee79f57554cb4d089a9cfb85070567ce. Latest consensus state: current_term: 1 leader_uuid: "ee79f57554cb4d089a9cfb85070567ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee79f57554cb4d089a9cfb85070567ce" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38555 } } }
I20260926 15:15:29.806368 11131 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee79f57554cb4d089a9cfb85070567ce [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:29.807902 11134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:15:29.811810 11134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:15:29.819679 11134 catalog_manager.cc:1347] Generated new cluster ID: 04c0f3caec0e4e5fa887ed808364a37e
I20260926 15:15:29.819768 11134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:15:29.836845 11134 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:15:29.838320 11134 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:15:29.855297 11134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee79f57554cb4d089a9cfb85070567ce: Generated new TSK 0
I20260926 15:15:29.856096 11134 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
I20260926 15:15:29.905612 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:38555
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:30.219244 11150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:30.219496 11150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:30.219641 11150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:30.226168 11150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:30.226300 11150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:30.226483 11150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:15:30.235551 11150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:38555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:30.236714 11150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:30.237900 11150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:30.249747 11158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:30.249711 11156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:30.249790 11155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:30.250556 11150 server_base.cc:1034] running on GCE node
I20260926 15:15:30.251106 11150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:30.251850 11150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:30.253033 11150 hybrid_clock.cc:648] HybridClock initialized: now 1790435730252997 us; error 24 us; skew 500 ppm
I20260926 15:15:30.253301 11150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:30 dist-test-slave-j6q7 krb5kdc[11045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435730, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:30.256242 11150 init.cc:377] Logged in from keytab as kudu/127.6.209.193@KRBTEST.COM (short username kudu)
I20260926 15:15:30.257346 11150 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:30.257467 11150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:30.257741 11150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:30.260262 11150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "dcb1434593224f6985b106cfb55dbca5"
format_stamp: "Formatted at 2026-09-26 15:15:30 on dist-test-slave-j6q7"
server_key: "dbc4219cf94f9edfcfcbc9396baa7166"
server_key_iv: "4e013ae98a6e68adaace31156643ddef"
server_key_version: "encryptionkey@0"
I20260926 15:15:30.261137 11150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "dcb1434593224f6985b106cfb55dbca5"
format_stamp: "Formatted at 2026-09-26 15:15:30 on dist-test-slave-j6q7"
server_key: "dbc4219cf94f9edfcfcbc9396baa7166"
server_key_iv: "4e013ae98a6e68adaace31156643ddef"
server_key_version: "encryptionkey@0"
I20260926 15:15:30.265594 11150 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:15:30.268615 11164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:30.269730 11150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:15:30.269917 11150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "dcb1434593224f6985b106cfb55dbca5"
format_stamp: "Formatted at 2026-09-26 15:15:30 on dist-test-slave-j6q7"
server_key: "dbc4219cf94f9edfcfcbc9396baa7166"
server_key_iv: "4e013ae98a6e68adaace31156643ddef"
server_key_version: "encryptionkey@0"
I20260926 15:15:30.270115 11150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:30.286082 11150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:30.289261 11150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:30.289510 11150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:30.290284 11150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:30.291435 11150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:30.291543 11150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:30.291671 11150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:30.291750 11150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:30.321590 11150 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:38533
I20260926 15:15:30.321659 11276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:38533 every 8 connection(s)
I20260926 15:15:30.322324 11150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:15:30.331758 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 11150
I20260926 15:15:30.331945 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:15:30.332499 6983 external_mini_cluster.cc:1428] Setting key f1ee0bb6d365b4f5e5e1e31341805b4c
Sep 26 15:15:30 dist-test-slave-j6q7 krb5kdc[11045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435730, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:30.340209 11279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:30.324215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:60163 (local address 127.6.209.254:38555)
0926 15:15:30.324578 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:30.324581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:30.325326 (+ 745us) server_negotiation.cc:407] Connection header received
0926 15:15:30.327068 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:30.327073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:30.327155 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:30.327262 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:30.328513 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:30.329184 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:30.330464 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:30.330801 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:30.330917 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:30.334083 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:30.334087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:30.334090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:30.334113 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:30.337428 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:30.338010 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:30.338017 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:30.338023 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:30.338152 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:30.338502 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:30.338508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:30.338514 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:30.338847 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:30.338997 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:30.339521 (+ 524us) server_negotiation.cc:299] Negotiation successful
0926 15:15:30.339763 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":195,"threads_started":1}
I20260926 15:15:30.340991 11280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:30.324544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:38555 (local address 127.6.209.193:60163)
0926 15:15:30.325182 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:30.325221 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:30.326815 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:30.327468 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:30.327475 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:30.327912 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:30.328371 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:30.328385 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:30.329313 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:30.329317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:30.330340 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:30.330350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:30.330987 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:30.330993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:30.331171 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:30.331773 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:30.331792 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:30.333910 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:30.337548 (+ 3638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:30.337553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:30.337563 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:30.337902 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:30.338247 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:30.338250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:30.338252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:30.338405 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:30.338956 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:30.338978 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:30.339206 (+ 228us) client_negotiation.cc:769] Sending connection context
0926 15:15:30.339410 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 15:15:30.339775 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":304,"threads_started":1}
I20260926 15:15:30.342351 11277 heartbeater.cc:347] Connected to a master server at 127.6.209.254:38555
I20260926 15:15:30.342610 11277 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:30.343135 11277 heartbeater.cc:511] Master 127.6.209.254:38555 requested a full tablet report, sending...
I20260926 15:15:30.344796 11089 ts_manager.cc:194] Registered new tserver with Master: dcb1434593224f6985b106cfb55dbca5 (127.6.209.193:38533)
I20260926 15:15:30.346150 11089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.193@KRBTEST.COM'} at 127.6.209.193:60163
WARNING: no policy specified for kudu/127.6.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
I20260926 15:15:30.371827 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:38555
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:30.663267 11283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:30.663491 11283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:30.663573 11283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:30.670217 11283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:30.670323 11283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:30.670424 11283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:15:30.678745 11283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:38555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:30.679768 11283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:30.680910 11283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:30.692823 11289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:30.692770 11288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:30.692863 11291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:30.693799 11283 server_base.cc:1034] running on GCE node
I20260926 15:15:30.694270 11283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:30.695051 11283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:30.696233 11283 hybrid_clock.cc:648] HybridClock initialized: now 1790435730696180 us; error 42 us; skew 500 ppm
I20260926 15:15:30.696519 11283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:30 dist-test-slave-j6q7 krb5kdc[11045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435730, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:30.699193 11283 init.cc:377] Logged in from keytab as kudu/127.6.209.194@KRBTEST.COM (short username kudu)
I20260926 15:15:30.700191 11283 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:30.700296 11283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:30.700635 11283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:30.703233 11283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "efb491af526f4677bbbfcffb1ae7242e"
format_stamp: "Formatted at 2026-09-26 15:15:30 on dist-test-slave-j6q7"
server_key: "b6427377725c937770f7a6e437795e10"
server_key_iv: "8cfef472d5793e851d217be6aa19faaa"
server_key_version: "encryptionkey@0"
I20260926 15:15:30.703996 11283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "efb491af526f4677bbbfcffb1ae7242e"
format_stamp: "Formatted at 2026-09-26 15:15:30 on dist-test-slave-j6q7"
server_key: "b6427377725c937770f7a6e437795e10"
server_key_iv: "8cfef472d5793e851d217be6aa19faaa"
server_key_version: "encryptionkey@0"
I20260926 15:15:30.708204 11283 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 15:15:30.711225 11297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:30.712318 11283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:15:30.712462 11283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "efb491af526f4677bbbfcffb1ae7242e"
format_stamp: "Formatted at 2026-09-26 15:15:30 on dist-test-slave-j6q7"
server_key: "b6427377725c937770f7a6e437795e10"
server_key_iv: "8cfef472d5793e851d217be6aa19faaa"
server_key_version: "encryptionkey@0"
I20260926 15:15:30.712661 11283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:30.744097 11283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:30.747692 11283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:30.747959 11283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:30.748791 11283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:30.750023 11283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:30.750106 11283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:30.750280 11283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:30.750357 11283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:30.787029 11283 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:38445
I20260926 15:15:30.787104 11409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:38445 every 8 connection(s)
I20260926 15:15:30.787765 11283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:15:30.796723 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 11283
I20260926 15:15:30.796900 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:15:30.797370 6983 external_mini_cluster.cc:1428] Setting key 9c68595d5876b95d5add8cce1d53743a
Sep 26 15:15:30 dist-test-slave-j6q7 krb5kdc[11045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435730, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:30.805692 11279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:30.790016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:57835 (local address 127.6.209.254:38555)
0926 15:15:30.790172 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:30.790177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:30.791192 (+ 1015us) server_negotiation.cc:407] Connection header received
0926 15:15:30.792711 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:30.792715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:30.792773 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:30.792915 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:30.794143 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:30.794772 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:30.796226 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:30.796862 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:30.797000 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:30.800271 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:30.800276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:30.800279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:30.800307 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:30.802785 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:30.803504 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:30.803509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:30.803512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:30.803626 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:30.804058 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:30.804064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:30.804066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:30.804340 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:30.804517 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:30.805187 (+ 670us) server_negotiation.cc:299] Negotiation successful
0926 15:15:30.805364 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 15:15:30.806648 11412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:30.790452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:38555 (local address 127.6.209.194:57835)
0926 15:15:30.791054 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:30.791087 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:30.792498 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:30.793090 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:30.793098 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:30.793535 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:30.794015 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:30.794030 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:30.794895 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:30.794898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:30.796087 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:30.796098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:30.797106 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:30.797111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:30.797277 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:30.797896 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:30.797915 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:30.800070 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:30.802944 (+ 2874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:30.802952 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:30.802965 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:30.803403 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:30.803759 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:30.803763 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:30.803767 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:30.803966 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:30.804503 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:30.804509 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:30.804791 (+ 282us) client_negotiation.cc:769] Sending connection context
0926 15:15:30.805116 (+ 325us) client_negotiation.cc:240] Negotiation successful
0926 15:15:30.805588 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":248,"threads_started":1}
I20260926 15:15:30.807898 11410 heartbeater.cc:347] Connected to a master server at 127.6.209.254:38555
I20260926 15:15:30.808197 11410 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:30.808928 11410 heartbeater.cc:511] Master 127.6.209.254:38555 requested a full tablet report, sending...
I20260926 15:15:30.810321 11089 ts_manager.cc:194] Registered new tserver with Master: efb491af526f4677bbbfcffb1ae7242e (127.6.209.194:38445)
I20260926 15:15:30.811400 11089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.194@KRBTEST.COM'} at 127.6.209.194:57835
WARNING: no policy specified for kudu/127.6.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
I20260926 15:15:30.837956 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:38555
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:31.125177 11415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:31.125401 11415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:31.125576 11415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:31.131839 11415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:31.131916 11415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:31.132040 11415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:15:31.140477 11415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:38555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:31.141665 11415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:31.142807 11415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:31.154145 11423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:31.154104 11420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:31.154177 11421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:31.155005 11415 server_base.cc:1034] running on GCE node
I20260926 15:15:31.155539 11415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:31.156359 11415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:31.157573 11415 hybrid_clock.cc:648] HybridClock initialized: now 1790435731157541 us; error 47 us; skew 500 ppm
I20260926 15:15:31.157841 11415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:31 dist-test-slave-j6q7 krb5kdc[11045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435731, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:31.160542 11415 init.cc:377] Logged in from keytab as kudu/127.6.209.195@KRBTEST.COM (short username kudu)
I20260926 15:15:31.161705 11415 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:31.161825 11415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:31.162129 11415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:31.164697 11415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "580203bfbf284781971e55ef541e871d"
format_stamp: "Formatted at 2026-09-26 15:15:31 on dist-test-slave-j6q7"
server_key: "84cb5e34d54ec48d6e7a5c877bd76842"
server_key_iv: "e2eedb2e14bc145eb472366a2cd5c962"
server_key_version: "encryptionkey@0"
I20260926 15:15:31.165490 11415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "580203bfbf284781971e55ef541e871d"
format_stamp: "Formatted at 2026-09-26 15:15:31 on dist-test-slave-j6q7"
server_key: "84cb5e34d54ec48d6e7a5c877bd76842"
server_key_iv: "e2eedb2e14bc145eb472366a2cd5c962"
server_key_version: "encryptionkey@0"
I20260926 15:15:31.169965 11415 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 15:15:31.172868 11429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:31.174127 11415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:15:31.174296 11415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "580203bfbf284781971e55ef541e871d"
format_stamp: "Formatted at 2026-09-26 15:15:31 on dist-test-slave-j6q7"
server_key: "84cb5e34d54ec48d6e7a5c877bd76842"
server_key_iv: "e2eedb2e14bc145eb472366a2cd5c962"
server_key_version: "encryptionkey@0"
I20260926 15:15:31.174463 11415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:31.195219 11415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:31.198923 11415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:31.199142 11415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:31.199892 11415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:31.201014 11415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:31.201095 11415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:31.201164 11415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:31.201217 11415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:31.229785 11415 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:35317
I20260926 15:15:31.229861 11541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:35317 every 8 connection(s)
I20260926 15:15:31.230592 11415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:15:31.232038 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 11415
I20260926 15:15:31.232192 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:15:31.232606 6983 external_mini_cluster.cc:1428] Setting key aee1741eff64eea7445076ad51fd4268
Sep 26 15:15:31 dist-test-slave-j6q7 krb5kdc[11045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435731, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:31.248533 11279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:31.232795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:54041 (local address 127.6.209.254:38555)
0926 15:15:31.233026 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:31.233032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:31.234306 (+ 1274us) server_negotiation.cc:407] Connection header received
0926 15:15:31.236344 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:31.236392 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:31.236504 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:31.236611 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:31.238031 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.238692 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.240043 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.240509 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.240595 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:31.243525 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:31.243528 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:31.243531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:31.243555 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:31.245882 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:31.246619 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:31.246622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:31.246624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:31.246804 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:31.247240 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:31.247244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:31.247247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:31.247554 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:31.247652 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:31.248181 (+ 529us) server_negotiation.cc:299] Negotiation successful
0926 15:15:31.248319 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 15:15:31.249372 11544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:31.233361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:38555 (local address 127.6.209.195:54041)
0926 15:15:31.234177 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:31.234218 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:31.236004 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:31.236864 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:31.236874 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:31.237410 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:31.237888 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:31.237903 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.238802 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.238805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:31.239865 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:31.239876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.240667 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.240671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:31.240888 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:31.241504 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:31.241524 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:31.243356 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:31.246031 (+ 2675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:31.246039 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:31.246054 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:31.246516 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:31.246943 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:31.246947 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:31.246951 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:31.247123 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:31.247653 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:31.247658 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:31.247913 (+ 255us) client_negotiation.cc:769] Sending connection context
0926 15:15:31.248105 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 15:15:31.248401 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":355,"threads_started":1}
I20260926 15:15:31.250563 11542 heartbeater.cc:347] Connected to a master server at 127.6.209.254:38555
I20260926 15:15:31.250819 11542 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:31.251355 11542 heartbeater.cc:511] Master 127.6.209.254:38555 requested a full tablet report, sending...
I20260926 15:15:31.252489 11089 ts_manager.cc:194] Registered new tserver with Master: 580203bfbf284781971e55ef541e871d (127.6.209.195:35317)
I20260926 15:15:31.253206 11089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.195@KRBTEST.COM'} at 127.6.209.195:54041
I20260926 15:15:31.259029 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:15:31.273494 11279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:31.263144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40962 (local address 127.6.209.254:38555)
0926 15:15:31.263331 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:31.263339 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:31.263721 (+ 382us) server_negotiation.cc:407] Connection header received
0926 15:15:31.263827 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:31.263830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:31.263879 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:31.263945 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:31.265241 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.265921 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.267053 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.267388 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.267462 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:31.269380 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:31.269384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:31.269387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:31.269426 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:31.271624 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:31.272092 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:31.272095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:31.272097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:31.272206 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:31.272460 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:31.272462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:31.272464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:31.272683 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:31.272878 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:31.273100 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 15:15:31.273224 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 15:15:31.276679 11089 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:15:31.282562 11089 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:15:31.314884 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:31.293407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41526 (local address 127.6.209.194:38445)
0926 15:15:31.307515 (+ 14108us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:31.307530 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:31.307563 (+ 33us) server_negotiation.cc:407] Connection header received
0926 15:15:31.307673 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:31.307679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:31.307877 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:31.308049 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:31.308600 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.310084 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.312783 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.313739 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.313879 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:31.314078 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:31.314323 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 15:15:31.314523 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13989,"thread_start_us":13872,"threads_started":1}
I20260926 15:15:31.314992 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:31.293122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:38445 (local address 127.0.0.1:41526)
0926 15:15:31.305691 (+ 12569us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:31.305740 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:31.305945 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:31.308049 (+ 2104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:31.308055 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:31.308078 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:31.308450 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:31.308462 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.310227 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.310234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:31.312636 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:31.312648 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.313922 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.313927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:31.314085 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:31.314138 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 15:15:31.314270 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 15:15:31.314521 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12249,"mutex_wait_us":10,"thread_start_us":12084,"threads_started":1}
I20260926 15:15:31.316246 11558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:31.293153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:35317 (local address 127.0.0.1:52294)
0926 15:15:31.306560 (+ 13407us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:31.306582 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:31.306704 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:31.309103 (+ 2399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:31.309108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:31.309128 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:31.309547 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:31.309559 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.310988 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.310994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:31.312959 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:31.312971 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.315445 (+ 2474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.315451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:31.315610 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:31.315641 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 15:15:31.315727 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 15:15:31.315840 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13268,"mutex_wait_us":85,"thread_start_us":17983,"threads_started":1}
I20260926 15:15:31.316558 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:31.293236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52294 (local address 127.6.209.195:35317)
0926 15:15:31.308600 (+ 15364us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:31.308606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:31.308631 (+ 25us) server_negotiation.cc:407] Connection header received
0926 15:15:31.308708 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:31.308714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:31.308929 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:31.309746 (+ 817us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:31.309983 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.310881 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.313128 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.314970 (+ 1842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.315085 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:31.315308 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:31.315788 (+ 480us) server_negotiation.cc:299] Negotiation successful
0926 15:15:31.315954 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15217,"thread_start_us":15084,"threads_started":1}
I20260926 15:15:31.318945 11345 tablet_service.cc:1467] Processing CreateTablet for tablet 39d1da017a084e6c9e3720530a54ea5c (DEFAULT_TABLE table=test-table [id=6431f9f0304345548641bfd0ab951666]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:31.320266 11345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d1da017a084e6c9e3720530a54ea5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:31.319885 11477 tablet_service.cc:1467] Processing CreateTablet for tablet 39d1da017a084e6c9e3720530a54ea5c (DEFAULT_TABLE table=test-table [id=6431f9f0304345548641bfd0ab951666]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:31.321477 11477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d1da017a084e6c9e3720530a54ea5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:31.322257 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:31.295663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37752 (local address 127.6.209.193:38533)
0926 15:15:31.307144 (+ 11481us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:31.307155 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:31.311540 (+ 4385us) server_negotiation.cc:407] Connection header received
0926 15:15:31.311671 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:31.311676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:31.311832 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:31.311998 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:31.312580 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.313973 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.319888 (+ 5915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.320894 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.321060 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:31.321232 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:31.321709 (+ 477us) server_negotiation.cc:299] Negotiation successful
0926 15:15:31.321880 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11350,"thread_start_us":11252,"threads_started":1}
I20260926 15:15:31.323071 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:31.295071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:38533 (local address 127.0.0.1:37752)
0926 15:15:31.311423 (+ 16352us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:31.311442 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:31.311545 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:31.312010 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:31.312014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:31.312035 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:31.312406 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:31.312419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.314138 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.314143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:31.319753 (+ 5610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:31.319763 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.321353 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.321369 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:31.321524 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:31.321551 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 15:15:31.322658 (+ 1107us) client_negotiation.cc:240] Negotiation successful
0926 15:15:31.322773 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16213,"mutex_wait_us":3,"thread_start_us":11209,"threads_started":1}
I20260926 15:15:31.326632 11212 tablet_service.cc:1467] Processing CreateTablet for tablet 39d1da017a084e6c9e3720530a54ea5c (DEFAULT_TABLE table=test-table [id=6431f9f0304345548641bfd0ab951666]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:31.327883 11212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d1da017a084e6c9e3720530a54ea5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:31.333844 11560 tablet_bootstrap.cc:492] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e: Bootstrap starting.
I20260926 15:15:31.335469 11561 tablet_bootstrap.cc:492] T 39d1da017a084e6c9e3720530a54ea5c P 580203bfbf284781971e55ef541e871d: Bootstrap starting.
I20260926 15:15:31.337142 11560 tablet_bootstrap.cc:654] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:31.337747 11562 tablet_bootstrap.cc:492] T 39d1da017a084e6c9e3720530a54ea5c P dcb1434593224f6985b106cfb55dbca5: Bootstrap starting.
I20260926 15:15:31.338260 11561 tablet_bootstrap.cc:654] T 39d1da017a084e6c9e3720530a54ea5c P 580203bfbf284781971e55ef541e871d: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:31.338374 11560 log.cc:824] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:31.339433 11561 log.cc:824] T 39d1da017a084e6c9e3720530a54ea5c P 580203bfbf284781971e55ef541e871d: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:31.340858 11562 tablet_bootstrap.cc:654] T 39d1da017a084e6c9e3720530a54ea5c P dcb1434593224f6985b106cfb55dbca5: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:31.340900 11560 tablet_bootstrap.cc:492] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e: No bootstrap required, opened a new log
I20260926 15:15:31.341158 11560 ts_tablet_manager.cc:1397] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 15:15:31.341876 11562 log.cc:824] T 39d1da017a084e6c9e3720530a54ea5c P dcb1434593224f6985b106cfb55dbca5: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:31.342180 11561 tablet_bootstrap.cc:492] T 39d1da017a084e6c9e3720530a54ea5c P 580203bfbf284781971e55ef541e871d: No bootstrap required, opened a new log
I20260926 15:15:31.342427 11561 ts_tablet_manager.cc:1397] T 39d1da017a084e6c9e3720530a54ea5c P 580203bfbf284781971e55ef541e871d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 15:15:31.343984 11562 tablet_bootstrap.cc:492] T 39d1da017a084e6c9e3720530a54ea5c P dcb1434593224f6985b106cfb55dbca5: No bootstrap required, opened a new log
I20260926 15:15:31.344213 11562 ts_tablet_manager.cc:1397] T 39d1da017a084e6c9e3720530a54ea5c P dcb1434593224f6985b106cfb55dbca5: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 15:15:31.348639 11277 heartbeater.cc:503] Master 127.6.209.254:38555 was elected leader, sending a full tablet report...
I20260926 15:15:31.350719 11560 raft_consensus.cc:348] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580203bfbf284781971e55ef541e871d" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35317 } } peers { permanent_uuid: "efb491af526f4677bbbfcffb1ae7242e" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 38445 } } peers { permanent_uuid: "dcb1434593224f6985b106cfb55dbca5" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38533 } }
I20260926 15:15:31.351089 11560 raft_consensus.cc:374] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:31.351166 11560 raft_consensus.cc:729] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb491af526f4677bbbfcffb1ae7242e, State: Initialized, Role: FOLLOWER
I20260926 15:15:31.351681 11562 raft_consensus.cc:348] T 39d1da017a084e6c9e3720530a54ea5c P dcb1434593224f6985b106cfb55dbca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580203bfbf284781971e55ef541e871d" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35317 } } peers { permanent_uuid: "efb491af526f4677bbbfcffb1ae7242e" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 38445 } } peers { permanent_uuid: "dcb1434593224f6985b106cfb55dbca5" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38533 } }
I20260926 15:15:31.351755 11560 consensus_queue.cc:260] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580203bfbf284781971e55ef541e871d" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35317 } } peers { permanent_uuid: "efb491af526f4677bbbfcffb1ae7242e" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 38445 } } peers { permanent_uuid: "dcb1434593224f6985b106cfb55dbca5" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38533 } }
I20260926 15:15:31.352042 11562 raft_consensus.cc:374] T 39d1da017a084e6c9e3720530a54ea5c P dcb1434593224f6985b106cfb55dbca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:31.352114 11562 raft_consensus.cc:729] T 39d1da017a084e6c9e3720530a54ea5c P dcb1434593224f6985b106cfb55dbca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcb1434593224f6985b106cfb55dbca5, State: Initialized, Role: FOLLOWER
I20260926 15:15:31.351899 11561 raft_consensus.cc:348] T 39d1da017a084e6c9e3720530a54ea5c P 580203bfbf284781971e55ef541e871d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580203bfbf284781971e55ef541e871d" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35317 } } peers { permanent_uuid: "efb491af526f4677bbbfcffb1ae7242e" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 38445 } } peers { permanent_uuid: "dcb1434593224f6985b106cfb55dbca5" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38533 } }
I20260926 15:15:31.352252 11561 raft_consensus.cc:374] T 39d1da017a084e6c9e3720530a54ea5c P 580203bfbf284781971e55ef541e871d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:31.352345 11561 raft_consensus.cc:729] T 39d1da017a084e6c9e3720530a54ea5c P 580203bfbf284781971e55ef541e871d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580203bfbf284781971e55ef541e871d, State: Initialized, Role: FOLLOWER
I20260926 15:15:31.352895 11562 consensus_queue.cc:260] T 39d1da017a084e6c9e3720530a54ea5c P dcb1434593224f6985b106cfb55dbca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580203bfbf284781971e55ef541e871d" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35317 } } peers { permanent_uuid: "efb491af526f4677bbbfcffb1ae7242e" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 38445 } } peers { permanent_uuid: "dcb1434593224f6985b106cfb55dbca5" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38533 } }
I20260926 15:15:31.353040 11561 consensus_queue.cc:260] T 39d1da017a084e6c9e3720530a54ea5c P 580203bfbf284781971e55ef541e871d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580203bfbf284781971e55ef541e871d" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35317 } } peers { permanent_uuid: "efb491af526f4677bbbfcffb1ae7242e" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 38445 } } peers { permanent_uuid: "dcb1434593224f6985b106cfb55dbca5" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38533 } }
I20260926 15:15:31.354947 11562 ts_tablet_manager.cc:1428] T 39d1da017a084e6c9e3720530a54ea5c P dcb1434593224f6985b106cfb55dbca5: Time spent starting tablet: real 0.011s user 0.005s sys 0.005s
I20260926 15:15:31.356226 11410 heartbeater.cc:503] Master 127.6.209.254:38555 was elected leader, sending a full tablet report...
I20260926 15:15:31.356628 11542 heartbeater.cc:503] Master 127.6.209.254:38555 was elected leader, sending a full tablet report...
I20260926 15:15:31.357303 11561 ts_tablet_manager.cc:1428] T 39d1da017a084e6c9e3720530a54ea5c P 580203bfbf284781971e55ef541e871d: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260926 15:15:31.358237 11560 ts_tablet_manager.cc:1428] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
I20260926 15:15:31.394989 11566 raft_consensus.cc:482] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:15:31.395217 11566 raft_consensus.cc:504] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580203bfbf284781971e55ef541e871d" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35317 } } peers { permanent_uuid: "efb491af526f4677bbbfcffb1ae7242e" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 38445 } } peers { permanent_uuid: "dcb1434593224f6985b106cfb55dbca5" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38533 } }
I20260926 15:15:31.396704 11566 leader_election.cc:288] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 580203bfbf284781971e55ef541e871d (127.6.209.195:35317), dcb1434593224f6985b106cfb55dbca5 (127.6.209.193:38533)
I20260926 15:15:31.402923 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:31.397160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:33479 (local address 127.6.209.195:35317)
0926 15:15:31.397356 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:31.397363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:31.398121 (+ 758us) server_negotiation.cc:407] Connection header received
0926 15:15:31.398302 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:31.398306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:31.398366 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:31.398467 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:31.398954 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.399622 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.401302 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.402053 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.402251 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:31.402319 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:31.402599 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 15:15:31.402704 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 15:15:31.402972 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:31.397144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:36719 (local address 127.6.209.193:38533)
0926 15:15:31.397356 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:31.397363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:31.398332 (+ 969us) server_negotiation.cc:407] Connection header received
0926 15:15:31.398477 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:31.398480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:31.398528 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:31.398629 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:31.399137 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.399716 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.401362 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.402103 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.402304 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:31.402364 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:31.402634 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 15:15:31.402741 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 15:15:31.403143 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:31.397078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:35317 (local address 127.6.209.194:33479)
0926 15:15:31.397987 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:31.398013 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:31.398157 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:31.398481 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:31.398484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:31.398505 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:31.398818 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:31.398826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.399722 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.399729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:31.401161 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:31.401172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.402289 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.402295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:31.402469 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:31.402489 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 15:15:31.402579 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 15:15:31.402693 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":51,"thread_start_us":521,"threads_started":1}
I20260926 15:15:31.403321 11570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:31.397418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:38533 (local address 127.6.209.194:36719)
0926 15:15:31.398241 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:31.398263 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:31.398357 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:31.398663 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:31.398666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:31.398691 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:31.399012 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:31.399018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.399835 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.399840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:31.401224 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:31.401235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.402323 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.402328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:31.402510 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:31.402529 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 15:15:31.402614 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 15:15:31.402714 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":66,"thread_start_us":496,"threads_started":1}
I20260926 15:15:31.404083 11232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d1da017a084e6c9e3720530a54ea5c" candidate_uuid: "efb491af526f4677bbbfcffb1ae7242e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcb1434593224f6985b106cfb55dbca5" is_pre_election: true
I20260926 15:15:31.404301 11497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d1da017a084e6c9e3720530a54ea5c" candidate_uuid: "efb491af526f4677bbbfcffb1ae7242e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "580203bfbf284781971e55ef541e871d" is_pre_election: true
I20260926 15:15:31.404670 11232 raft_consensus.cc:2454] T 39d1da017a084e6c9e3720530a54ea5c P dcb1434593224f6985b106cfb55dbca5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb491af526f4677bbbfcffb1ae7242e in term 0.
I20260926 15:15:31.404785 11497 raft_consensus.cc:2454] T 39d1da017a084e6c9e3720530a54ea5c P 580203bfbf284781971e55ef541e871d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb491af526f4677bbbfcffb1ae7242e in term 0.
I20260926 15:15:31.405324 11298 leader_election.cc:302] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dcb1434593224f6985b106cfb55dbca5, efb491af526f4677bbbfcffb1ae7242e; no voters:
I20260926 15:15:31.405694 11566 raft_consensus.cc:2781] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:15:31.405846 11566 raft_consensus.cc:482] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:15:31.405956 11566 raft_consensus.cc:3037] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:31.408061 11566 raft_consensus.cc:504] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580203bfbf284781971e55ef541e871d" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35317 } } peers { permanent_uuid: "efb491af526f4677bbbfcffb1ae7242e" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 38445 } } peers { permanent_uuid: "dcb1434593224f6985b106cfb55dbca5" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38533 } }
I20260926 15:15:31.408625 11566 leader_election.cc:288] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e [CANDIDATE]: Term 1 election: Requested vote from peers 580203bfbf284781971e55ef541e871d (127.6.209.195:35317), dcb1434593224f6985b106cfb55dbca5 (127.6.209.193:38533)
I20260926 15:15:31.408936 11497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d1da017a084e6c9e3720530a54ea5c" candidate_uuid: "efb491af526f4677bbbfcffb1ae7242e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "580203bfbf284781971e55ef541e871d"
I20260926 15:15:31.409102 11497 raft_consensus.cc:3037] T 39d1da017a084e6c9e3720530a54ea5c P 580203bfbf284781971e55ef541e871d [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:31.409083 11232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d1da017a084e6c9e3720530a54ea5c" candidate_uuid: "efb491af526f4677bbbfcffb1ae7242e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcb1434593224f6985b106cfb55dbca5"
I20260926 15:15:31.409274 11232 raft_consensus.cc:3037] T 39d1da017a084e6c9e3720530a54ea5c P dcb1434593224f6985b106cfb55dbca5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:31.411367 11497 raft_consensus.cc:2454] T 39d1da017a084e6c9e3720530a54ea5c P 580203bfbf284781971e55ef541e871d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb491af526f4677bbbfcffb1ae7242e in term 1.
I20260926 15:15:31.411734 11232 raft_consensus.cc:2454] T 39d1da017a084e6c9e3720530a54ea5c P dcb1434593224f6985b106cfb55dbca5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb491af526f4677bbbfcffb1ae7242e in term 1.
I20260926 15:15:31.411746 11299 leader_election.cc:302] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 580203bfbf284781971e55ef541e871d, efb491af526f4677bbbfcffb1ae7242e; no voters:
I20260926 15:15:31.412039 11566 raft_consensus.cc:2781] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:31.412427 11566 raft_consensus.cc:686] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e [term 1 LEADER]: Becoming Leader. State: Replica: efb491af526f4677bbbfcffb1ae7242e, State: Running, Role: LEADER
I20260926 15:15:31.412916 11566 consensus_queue.cc:237] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580203bfbf284781971e55ef541e871d" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35317 } } peers { permanent_uuid: "efb491af526f4677bbbfcffb1ae7242e" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 38445 } } peers { permanent_uuid: "dcb1434593224f6985b106cfb55dbca5" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38533 } }
I20260926 15:15:31.417001 11089 catalog_manager.cc:5495] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e reported cstate change: term changed from 0 to 1, leader changed from <none> to efb491af526f4677bbbfcffb1ae7242e (127.6.209.194). New cstate: current_term: 1 leader_uuid: "efb491af526f4677bbbfcffb1ae7242e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580203bfbf284781971e55ef541e871d" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb491af526f4677bbbfcffb1ae7242e" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 38445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcb1434593224f6985b106cfb55dbca5" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38533 } health_report { overall_health: UNKNOWN } } }
I20260926 15:15:31.445451 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:31.441112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41536 (local address 127.6.209.194:38445)
0926 15:15:31.441269 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:31.441273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:31.441291 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:15:31.441455 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:31.441459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:31.441529 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:31.441651 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:15:31.441944 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.442573 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.443621 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.443975 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.444063 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:31.444394 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:15:31.444955 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:15:31.445064 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:31.445110 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 15:15:31.445163 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 15:15:31.452942 11232 raft_consensus.cc:1261] T 39d1da017a084e6c9e3720530a54ea5c P dcb1434593224f6985b106cfb55dbca5 [term 1 FOLLOWER]: Refusing update from remote peer efb491af526f4677bbbfcffb1ae7242e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:31.452944 11497 raft_consensus.cc:1261] T 39d1da017a084e6c9e3720530a54ea5c P 580203bfbf284781971e55ef541e871d [term 1 FOLLOWER]: Refusing update from remote peer efb491af526f4677bbbfcffb1ae7242e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:31.453938 11571 consensus_queue.cc:1035] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcb1434593224f6985b106cfb55dbca5" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:31.454284 11566 consensus_queue.cc:1035] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e [LEADER]: Connected to new peer: Peer: permanent_uuid: "580203bfbf284781971e55ef541e871d" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:31.463593 11574 mvcc.cc:205] Tried to move back new op lower bound from 7333624756020895744 to 7333624755870666752. Current Snapshot: MvccSnapshot[applied={T|T < 7333624756020895744}]
I20260926 15:15:31.464828 11576 mvcc.cc:205] Tried to move back new op lower bound from 7333624756020895744 to 7333624755870666752. Current Snapshot: MvccSnapshot[applied={T|T < 7333624756020895744}]
I20260926 15:15:31.469153 11575 mvcc.cc:205] Tried to move back new op lower bound from 7333624756020895744 to 7333624755870666752. Current Snapshot: MvccSnapshot[applied={T|T < 7333624756020895744}]
I20260926 15:15:31.477020 11089 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:31.477286 11089 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:31.480247 11089 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee79f57554cb4d089a9cfb85070567ce: Sending DeleteTablet for 3 replicas of tablet 39d1da017a084e6c9e3720530a54ea5c
I20260926 15:15:31.481261 11477 tablet_service.cc:1514] Processing DeleteTablet for tablet 39d1da017a084e6c9e3720530a54ea5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:31 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:52294
I20260926 15:15:31.481261 11345 tablet_service.cc:1514] Processing DeleteTablet for tablet 39d1da017a084e6c9e3720530a54ea5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:31 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:41526
I20260926 15:15:31.482192 11212 tablet_service.cc:1514] Processing DeleteTablet for tablet 39d1da017a084e6c9e3720530a54ea5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:31 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:37752
I20260926 15:15:31.482975 11586 tablet_replica.cc:330] T 39d1da017a084e6c9e3720530a54ea5c P dcb1434593224f6985b106cfb55dbca5: stopping tablet replica
I20260926 15:15:31.483225 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 11150
I20260926 15:15:31.483461 11586 raft_consensus.cc:2229] T 39d1da017a084e6c9e3720530a54ea5c P dcb1434593224f6985b106cfb55dbca5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:31.483949 11586 raft_consensus.cc:2258] T 39d1da017a084e6c9e3720530a54ea5c P dcb1434593224f6985b106cfb55dbca5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:31.485724 11586 ts_tablet_manager.cc:1905] T 39d1da017a084e6c9e3720530a54ea5c P dcb1434593224f6985b106cfb55dbca5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:31.488931 11584 tablet_replica.cc:330] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e: stopping tablet replica
I20260926 15:15:31.489408 11585 tablet_replica.cc:330] T 39d1da017a084e6c9e3720530a54ea5c P 580203bfbf284781971e55ef541e871d: stopping tablet replica
I20260926 15:15:31.489733 11585 raft_consensus.cc:2229] T 39d1da017a084e6c9e3720530a54ea5c P 580203bfbf284781971e55ef541e871d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:31.490180 11585 raft_consensus.cc:2258] T 39d1da017a084e6c9e3720530a54ea5c P 580203bfbf284781971e55ef541e871d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:31.490659 11584 raft_consensus.cc:2229] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:15:31.490972 11586 ts_tablet_manager.cc:1918] T 39d1da017a084e6c9e3720530a54ea5c P dcb1434593224f6985b106cfb55dbca5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:15:31.491120 11586 log.cc:1196] T 39d1da017a084e6c9e3720530a54ea5c P dcb1434593224f6985b106cfb55dbca5: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-0/wal/wals/39d1da017a084e6c9e3720530a54ea5c
I20260926 15:15:31.491218 11584 raft_consensus.cc:2258] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:31.491619 11586 ts_tablet_manager.cc:1939] T 39d1da017a084e6c9e3720530a54ea5c P dcb1434593224f6985b106cfb55dbca5: Deleting consensus metadata
I20260926 15:15:31.492525 11585 ts_tablet_manager.cc:1905] T 39d1da017a084e6c9e3720530a54ea5c P 580203bfbf284781971e55ef541e871d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:31.492933 11584 ts_tablet_manager.cc:1905] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:31.493188 11073 catalog_manager.cc:4841] TS dcb1434593224f6985b106cfb55dbca5 (127.6.209.193:38533): tablet 39d1da017a084e6c9e3720530a54ea5c (table test-table [id=6431f9f0304345548641bfd0ab951666]) successfully deleted
I20260926 15:15:31.497617 11584 ts_tablet_manager.cc:1918] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:15:31.497768 11584 log.cc:1196] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/39d1da017a084e6c9e3720530a54ea5c
I20260926 15:15:31.498298 11584 ts_tablet_manager.cc:1939] T 39d1da017a084e6c9e3720530a54ea5c P efb491af526f4677bbbfcffb1ae7242e: Deleting consensus metadata
I20260926 15:15:31.499385 11585 ts_tablet_manager.cc:1918] T 39d1da017a084e6c9e3720530a54ea5c P 580203bfbf284781971e55ef541e871d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:15:31.499531 11585 log.cc:1196] T 39d1da017a084e6c9e3720530a54ea5c P 580203bfbf284781971e55ef541e871d: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/39d1da017a084e6c9e3720530a54ea5c
I20260926 15:15:31.499792 11075 catalog_manager.cc:4841] TS efb491af526f4677bbbfcffb1ae7242e (127.6.209.194:38445): tablet 39d1da017a084e6c9e3720530a54ea5c (table test-table [id=6431f9f0304345548641bfd0ab951666]) successfully deleted
I20260926 15:15:31.500011 11585 ts_tablet_manager.cc:1939] T 39d1da017a084e6c9e3720530a54ea5c P 580203bfbf284781971e55ef541e871d: Deleting consensus metadata
I20260926 15:15:31.501546 11074 catalog_manager.cc:4841] TS 580203bfbf284781971e55ef541e871d (127.6.209.195:35317): tablet 39d1da017a084e6c9e3720530a54ea5c (table test-table [id=6431f9f0304345548641bfd0ab951666]) successfully deleted
Sep 26 15:15:31 dist-test-slave-j6q7 krb5kdc[11045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.193@KRBTEST.COM
I20260926 15:15:31.511400 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:31.484574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37762 (local address 127.6.209.193:38533)
0926 15:15:31.484707 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:31.484713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:31.497314 (+ 12601us) server_negotiation.cc:407] Connection header received
0926 15:15:31.497610 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:31.497617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:31.497695 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:31.497769 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:31.499639 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.500487 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.501854 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.502247 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.502380 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:31.506162 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:31.506173 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:31.506187 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:31.506219 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:31.508744 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:31.509512 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:31.509518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:31.509519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:31.509621 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:31.510019 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:31.510026 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:31.510029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:31.510380 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:31.510594 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:31.510984 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 15:15:31.511129 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 15:15:31.511986 11272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:31.615239 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 11150
I20260926 15:15:31.631301 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 11283
Sep 26 15:15:31 dist-test-slave-j6q7 krb5kdc[11045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.194@KRBTEST.COM
I20260926 15:15:31.645368 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:31.632067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36704 (local address 127.6.209.194:38445)
0926 15:15:31.632236 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:31.632243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:31.632264 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:15:31.632471 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:31.632478 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:31.632546 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:31.632647 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:31.633858 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.634607 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.635812 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.636263 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.636396 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:31.640200 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:31.640216 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:31.640230 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:31.640265 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:31.643029 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:31.643582 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:31.643592 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:31.643599 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:31.643729 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:31.644047 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:31.644052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:31.644055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:31.644374 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:31.644658 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:31.644878 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 15:15:31.645080 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 15:15:31.646062 11405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:31.749013 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 11283
I20260926 15:15:31.765532 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 11415
Sep 26 15:15:31 dist-test-slave-j6q7 krb5kdc[11045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.195@KRBTEST.COM
I20260926 15:15:31.780217 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:31.766383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33798 (local address 127.6.209.195:35317)
0926 15:15:31.766573 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:31.766577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:31.766595 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:15:31.766744 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:31.766749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:31.766824 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:31.766923 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:31.768254 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.768968 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.770542 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:31.770856 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:31.770936 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:31.775107 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:31.775118 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:31.775132 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:31.775159 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:31.777945 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:31.778544 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:31.778550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:31.778552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:31.778684 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:31.779013 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:31.779017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:31.779020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:31.779308 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:31.779565 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:31.779775 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 15:15:31.779947 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 15:15:31.780802 11537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:31.862295 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 11415
I20260926 15:15:31.878297 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 11058
I20260926 15:15:31.879360 11119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:31.975164 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 11058
2026-09-26T15:15:31Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3715 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T15:15:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:15:32Z Disabled control of system clock
I20260926 15:15:32.035629 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:33253
--webserver_interface=127.6.209.254
--webserver_port=0
--builtin_ntp_servers=127.6.209.212:38809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:33253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 15:15:32.317250 11595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:32.317483 11595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:32.317615 11595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:32.323800 11595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:15:32.323873 11595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:32.323904 11595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:32.323943 11595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:15:32.324020 11595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:15:32.331576 11595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:38809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:33253
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:33253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:32.332645 11595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:32.334262 11595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:32.344842 11601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:32.344909 11603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:32.344887 11600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:32.345523 11595 server_base.cc:1034] running on GCE node
I20260926 15:15:32.346268 11595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:32.347476 11595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:32.348661 11595 hybrid_clock.cc:648] HybridClock initialized: now 1790435732348642 us; error 36 us; skew 500 ppm
I20260926 15:15:32.348994 11595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:15:32.356367 11595 webserver.cc:460] Webserver started at http://127.6.209.254:38315/ using document root <none> and password file <none>
I20260926 15:15:32.357071 11595 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:32.357172 11595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:32.357470 11595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:32.359989 11595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "1814e5d2cfb3432782bbe5f55e832dde"
format_stamp: "Formatted at 2026-09-26 15:15:32 on dist-test-slave-j6q7"
server_key: "f4ce30eff8277b53ca37163a14c910c8"
server_key_iv: "d505769be3117136b24304a21eec3be4"
server_key_version: "encryptionkey@0"
I20260926 15:15:32.360718 11595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "1814e5d2cfb3432782bbe5f55e832dde"
format_stamp: "Formatted at 2026-09-26 15:15:32 on dist-test-slave-j6q7"
server_key: "f4ce30eff8277b53ca37163a14c910c8"
server_key_iv: "d505769be3117136b24304a21eec3be4"
server_key_version: "encryptionkey@0"
I20260926 15:15:32.365304 11595 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:15:32.368376 11609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:32.369549 11595 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:15:32.369704 11595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "1814e5d2cfb3432782bbe5f55e832dde"
format_stamp: "Formatted at 2026-09-26 15:15:32 on dist-test-slave-j6q7"
server_key: "f4ce30eff8277b53ca37163a14c910c8"
server_key_iv: "d505769be3117136b24304a21eec3be4"
server_key_version: "encryptionkey@0"
I20260926 15:15:32.369884 11595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:32.375006 11595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:32.375631 11595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:32.375949 11595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:32.392488 11595 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:33253
I20260926 15:15:32.392515 11660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:33253 every 8 connection(s)
I20260926 15:15:32.393808 11595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:15:32.397147 11661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:32.399148 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 11595
I20260926 15:15:32.399326 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:15:32.400050 6983 external_mini_cluster.cc:1428] Setting key dee41ac5d20d5179e01d3c103ee33ae2
I20260926 15:15:32.407526 11664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:32.402018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58236 (local address 127.6.209.254:33253)
0926 15:15:32.402913 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:32.402923 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:32.402958 (+ 35us) server_negotiation.cc:407] Connection header received
0926 15:15:32.404308 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:32.404326 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:32.404650 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:32.405028 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:32.406060 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:32.406065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:32.406079 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:15:32.406094 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:32.406141 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:15:32.406154 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:32.406395 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:32.406525 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 15:15:32.406738 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":335,"threads_started":1}
I20260926 15:15:32.408083 11661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde: Bootstrap starting.
I20260926 15:15:32.410696 11661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:32.411597 11661 log.cc:824] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:32.414034 11661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde: No bootstrap required, opened a new log
I20260926 15:15:32.420231 11661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1814e5d2cfb3432782bbe5f55e832dde" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 33253 } }
I20260926 15:15:32.420531 11661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:32.420621 11661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1814e5d2cfb3432782bbe5f55e832dde, State: Initialized, Role: FOLLOWER
I20260926 15:15:32.421167 11661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1814e5d2cfb3432782bbe5f55e832dde" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 33253 } }
I20260926 15:15:32.421375 11661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:15:32.421501 11661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:15:32.421648 11661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:32.423377 11661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1814e5d2cfb3432782bbe5f55e832dde" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 33253 } }
I20260926 15:15:32.423785 11661 leader_election.cc:302] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1814e5d2cfb3432782bbe5f55e832dde; no voters:
I20260926 15:15:32.424268 11661 leader_election.cc:288] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:15:32.424381 11666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:32.425057 11666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde [term 1 LEADER]: Becoming Leader. State: Replica: 1814e5d2cfb3432782bbe5f55e832dde, State: Running, Role: LEADER
I20260926 15:15:32.425576 11666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1814e5d2cfb3432782bbe5f55e832dde" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 33253 } }
I20260926 15:15:32.426083 11661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:15:32.430892 11668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1814e5d2cfb3432782bbe5f55e832dde. Latest consensus state: current_term: 1 leader_uuid: "1814e5d2cfb3432782bbe5f55e832dde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1814e5d2cfb3432782bbe5f55e832dde" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 33253 } } }
I20260926 15:15:32.430845 11667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1814e5d2cfb3432782bbe5f55e832dde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1814e5d2cfb3432782bbe5f55e832dde" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 33253 } } }
I20260926 15:15:32.431197 11667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:32.431195 11668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde [sys.catalog]: This master's current role is: LEADER
W20260926 15:15:32.436058 11680 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:15:32.436172 11680 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:15:32.436295 11681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:15:32.437783 11681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:15:32.444636 11681 catalog_manager.cc:1347] Generated new cluster ID: 9842e6b6d00448bda9523fe14cc2cb35
I20260926 15:15:32.444729 11681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:15:32.473169 11681 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:15:32.473491 11681 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:15:32.478535 11681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde: Generated new TSK 0
I20260926 15:15:32.479149 11681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:15:32.486795 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:33253
--builtin_ntp_servers=127.6.209.212:38809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 15:15:32.791379 11685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:32.791620 11685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:32.791697 11685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:32.798259 11685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:32.798362 11685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:32.798525 11685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:15:32.806377 11685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:38809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:33253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:32.807495 11685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:32.809067 11685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:32.820856 11691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:32.820874 11690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:32.820860 11693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:32.821895 11685 server_base.cc:1034] running on GCE node
I20260926 15:15:32.822463 11685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:32.823201 11685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:32.824407 11685 hybrid_clock.cc:648] HybridClock initialized: now 1790435732824374 us; error 32 us; skew 500 ppm
I20260926 15:15:32.824683 11685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:15:32.826722 11685 webserver.cc:460] Webserver started at http://127.6.209.193:39687/ using document root <none> and password file <none>
I20260926 15:15:32.827433 11685 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:32.827545 11685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:32.827816 11685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:32.830689 11685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "4847db036a46433f8926f218729de1d4"
format_stamp: "Formatted at 2026-09-26 15:15:32 on dist-test-slave-j6q7"
server_key: "a46fc1a932737141ae25336cbf64f600"
server_key_iv: "419ceb899adef42eff63508f74da5546"
server_key_version: "encryptionkey@0"
I20260926 15:15:32.831413 11685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "4847db036a46433f8926f218729de1d4"
format_stamp: "Formatted at 2026-09-26 15:15:32 on dist-test-slave-j6q7"
server_key: "a46fc1a932737141ae25336cbf64f600"
server_key_iv: "419ceb899adef42eff63508f74da5546"
server_key_version: "encryptionkey@0"
I20260926 15:15:32.835973 11685 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260926 15:15:32.839052 11699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:32.840166 11685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:15:32.840322 11685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "4847db036a46433f8926f218729de1d4"
format_stamp: "Formatted at 2026-09-26 15:15:32 on dist-test-slave-j6q7"
server_key: "a46fc1a932737141ae25336cbf64f600"
server_key_iv: "419ceb899adef42eff63508f74da5546"
server_key_version: "encryptionkey@0"
I20260926 15:15:32.840524 11685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:32.845947 11685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:32.846647 11685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:32.846912 11685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:32.847673 11685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:32.848773 11685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:32.848898 11685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:32.849009 11685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:32.849086 11685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:32.878818 11685 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:38111
I20260926 15:15:32.878899 11811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:38111 every 8 connection(s)
I20260926 15:15:32.880244 11685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:15:32.884343 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 11685
I20260926 15:15:32.884526 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:15:32.885147 6983 external_mini_cluster.cc:1428] Setting key 8e45eb8318595b6b840f1946954edc2a
I20260926 15:15:32.887665 11664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:32.882815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:36229 (local address 127.6.209.254:33253)
0926 15:15:32.882982 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:32.882986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:32.884085 (+ 1099us) server_negotiation.cc:407] Connection header received
0926 15:15:32.885722 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:32.885752 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:32.885815 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:32.885933 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:32.887067 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:32.887072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:32.887076 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:15:32.887090 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:32.887126 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:15:32.887136 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:32.887236 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:32.887394 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 15:15:32.887449 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:15:32.887735 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:33253
--builtin_ntp_servers=127.6.209.212:38809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 15:15:32.888403 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:32.883186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:33253 (local address 127.6.209.193:36229)
0926 15:15:32.883961 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:32.883990 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:32.885498 (+ 1508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:32.886134 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:32.886142 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:32.886221 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:32.886789 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:15:32.886791 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:32.886823 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:15:32.886918 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:32.887215 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:32.887222 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:32.887228 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 15:15:32.887458 (+ 230us) client_negotiation.cc:240] Negotiation successful
0926 15:15:32.887646 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":237,"threads_started":1}
I20260926 15:15:32.889771 11812 heartbeater.cc:347] Connected to a master server at 127.6.209.254:33253
I20260926 15:15:32.889993 11812 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:32.890409 11812 heartbeater.cc:511] Master 127.6.209.254:33253 requested a full tablet report, sending...
I20260926 15:15:32.892192 11626 ts_manager.cc:194] Registered new tserver with Master: 4847db036a46433f8926f218729de1d4 (127.6.209.193:38111)
W20260926 15:15:33.172377 11815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:33.172652 11815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:33.172794 11815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:33.179026 11815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:33.179095 11815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:33.179176 11815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:15:33.186949 11815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:38809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:33253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:33.187870 11815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:33.189263 11815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:33.200778 11820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:33.200747 11823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:33.200838 11821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:33.201429 11815 server_base.cc:1034] running on GCE node
I20260926 15:15:33.201902 11815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:33.202685 11815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:33.203917 11815 hybrid_clock.cc:648] HybridClock initialized: now 1790435733203880 us; error 42 us; skew 500 ppm
I20260926 15:15:33.204166 11815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:15:33.206111 11815 webserver.cc:460] Webserver started at http://127.6.209.194:40619/ using document root <none> and password file <none>
I20260926 15:15:33.206705 11815 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:33.206795 11815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:33.207068 11815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:33.209484 11815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "90a2d39d31d6466887087f219aa03aa6"
format_stamp: "Formatted at 2026-09-26 15:15:33 on dist-test-slave-j6q7"
server_key: "a252c9a87193aaebffd067eae7eb3e62"
server_key_iv: "8f302134839dc4b33dc96760520f8e71"
server_key_version: "encryptionkey@0"
I20260926 15:15:33.210139 11815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "90a2d39d31d6466887087f219aa03aa6"
format_stamp: "Formatted at 2026-09-26 15:15:33 on dist-test-slave-j6q7"
server_key: "a252c9a87193aaebffd067eae7eb3e62"
server_key_iv: "8f302134839dc4b33dc96760520f8e71"
server_key_version: "encryptionkey@0"
I20260926 15:15:33.214551 11815 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 15:15:33.217561 11829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:33.218631 11815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:15:33.218789 11815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "90a2d39d31d6466887087f219aa03aa6"
format_stamp: "Formatted at 2026-09-26 15:15:33 on dist-test-slave-j6q7"
server_key: "a252c9a87193aaebffd067eae7eb3e62"
server_key_iv: "8f302134839dc4b33dc96760520f8e71"
server_key_version: "encryptionkey@0"
I20260926 15:15:33.218986 11815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:33.224445 11815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:33.225167 11815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:33.225425 11815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:33.226431 11815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:33.227800 11815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:33.227880 11815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:33.228035 11815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:33.228094 11815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:33.258497 11815 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:45289
I20260926 15:15:33.258548 11941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:45289 every 8 connection(s)
I20260926 15:15:33.259560 11815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:15:33.263895 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 11815
I20260926 15:15:33.264055 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:15:33.264554 6983 external_mini_cluster.cc:1428] Setting key 8878e3825bb980c1d5fa4dc0cdc11448
I20260926 15:15:33.267388 11664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:33.261687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:40135 (local address 127.6.209.254:33253)
0926 15:15:33.261851 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:33.261856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:33.263108 (+ 1252us) server_negotiation.cc:407] Connection header received
0926 15:15:33.265224 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:33.265229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:33.265298 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:33.265409 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:33.266769 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:33.266773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:33.266777 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:15:33.266791 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:33.266820 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:15:33.266831 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:33.266920 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:33.267153 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 15:15:33.267195 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 15:15:33.267375 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:33253
--builtin_ntp_servers=127.6.209.212:38809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 15:15:33.268643 11944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:33.262179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:33253 (local address 127.6.209.194:40135)
0926 15:15:33.262974 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:33.263009 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:33.264945 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:33.265607 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:33.265615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:33.265696 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:33.266450 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:15:33.266454 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:33.266500 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:15:33.266602 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:33.266931 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:33.266940 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:33.266960 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 15:15:33.267225 (+ 265us) client_negotiation.cc:240] Negotiation successful
0926 15:15:33.267446 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":254,"threads_started":1}
I20260926 15:15:33.269936 11942 heartbeater.cc:347] Connected to a master server at 127.6.209.254:33253
I20260926 15:15:33.270201 11942 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:33.270814 11942 heartbeater.cc:511] Master 127.6.209.254:33253 requested a full tablet report, sending...
I20260926 15:15:33.272135 11626 ts_manager.cc:194] Registered new tserver with Master: 90a2d39d31d6466887087f219aa03aa6 (127.6.209.194:45289)
W20260926 15:15:33.567028 11945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:33.567348 11945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:33.567509 11945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:33.574287 11945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:33.574393 11945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:33.574573 11945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:15:33.582801 11945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:38809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:33253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:33.583886 11945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:33.585431 11945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:33.596968 11950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:33.597018 11951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:33.597072 11953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:33.597591 11945 server_base.cc:1034] running on GCE node
I20260926 15:15:33.598062 11945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:33.598794 11945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:33.600003 11945 hybrid_clock.cc:648] HybridClock initialized: now 1790435733599968 us; error 41 us; skew 500 ppm
I20260926 15:15:33.600314 11945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:15:33.602391 11945 webserver.cc:460] Webserver started at http://127.6.209.195:41905/ using document root <none> and password file <none>
I20260926 15:15:33.602974 11945 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:33.603101 11945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:33.603380 11945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:33.605940 11945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "404955450036498b95338c11b534a4d8"
format_stamp: "Formatted at 2026-09-26 15:15:33 on dist-test-slave-j6q7"
server_key: "a9c5ef5fc186dcdb226832f9d2ad7c3e"
server_key_iv: "569ae617b8ef0cc73627b895673d070a"
server_key_version: "encryptionkey@0"
I20260926 15:15:33.606669 11945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "404955450036498b95338c11b534a4d8"
format_stamp: "Formatted at 2026-09-26 15:15:33 on dist-test-slave-j6q7"
server_key: "a9c5ef5fc186dcdb226832f9d2ad7c3e"
server_key_iv: "569ae617b8ef0cc73627b895673d070a"
server_key_version: "encryptionkey@0"
I20260926 15:15:33.610934 11945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 15:15:33.613920 11959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:33.615098 11945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:15:33.615270 11945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "404955450036498b95338c11b534a4d8"
format_stamp: "Formatted at 2026-09-26 15:15:33 on dist-test-slave-j6q7"
server_key: "a9c5ef5fc186dcdb226832f9d2ad7c3e"
server_key_iv: "569ae617b8ef0cc73627b895673d070a"
server_key_version: "encryptionkey@0"
I20260926 15:15:33.615469 11945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:33.620606 11945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:33.621366 11945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:33.621615 11945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:33.622475 11945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:33.623661 11945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:33.623775 11945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:33.623889 11945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:33.623963 11945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:33.654310 11945 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:37747
I20260926 15:15:33.654383 12071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:37747 every 8 connection(s)
I20260926 15:15:33.655493 11945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:15:33.662586 11664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:33.657871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:48801 (local address 127.6.209.254:33253)
0926 15:15:33.658036 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:33.658040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:33.659272 (+ 1232us) server_negotiation.cc:407] Connection header received
0926 15:15:33.660655 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:33.660659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:33.660704 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:33.660785 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:33.662032 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:33.662036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:33.662039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:15:33.662053 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:33.662081 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:15:33.662092 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:33.662210 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:33.662375 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 15:15:33.662417 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 15:15:33.663424 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:33.658349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:33253 (local address 127.6.209.195:48801)
0926 15:15:33.659125 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:33.659156 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:33.660461 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:33.661034 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:33.661042 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:33.661123 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:33.661777 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:15:33.661780 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:33.661810 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:15:33.661883 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:33.662195 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:33.662201 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:33.662205 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:15:33.662425 (+ 220us) client_negotiation.cc:240] Negotiation successful
0926 15:15:33.662576 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":228,"threads_started":1}
I20260926 15:15:33.664341 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 11945
I20260926 15:15:33.664527 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:15:33.664726 12072 heartbeater.cc:347] Connected to a master server at 127.6.209.254:33253
I20260926 15:15:33.664963 6983 external_mini_cluster.cc:1428] Setting key 83efc575ebacf6f1084218d3f8875614
I20260926 15:15:33.664986 12072 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:33.665555 12072 heartbeater.cc:511] Master 127.6.209.254:33253 requested a full tablet report, sending...
I20260926 15:15:33.666975 11626 ts_manager.cc:194] Registered new tserver with Master: 404955450036498b95338c11b534a4d8 (127.6.209.195:37747)
I20260926 15:15:33.667900 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:15:33.673566 11664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:33.671588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58252 (local address 127.6.209.254:33253)
0926 15:15:33.671752 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:33.671759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:33.672141 (+ 382us) server_negotiation.cc:407] Connection header received
0926 15:15:33.672241 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:33.672245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:33.672323 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:33.672390 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:33.673145 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:33.673148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:33.673150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:15:33.673162 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:33.673184 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:15:33.673193 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:33.673272 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:33.673315 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 15:15:33.673346 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 15:15:33.677101 11625 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:15:33.682905 11625 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:15:33.716441 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:33.693057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52598 (local address 127.6.209.195:37747)
0926 15:15:33.713229 (+ 20172us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:33.713239 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:33.713272 (+ 33us) server_negotiation.cc:407] Connection header received
0926 15:15:33.713329 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:33.713337 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:33.713523 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:33.713660 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:33.715565 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:33.715571 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:33.715579 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:15:33.715593 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:33.715628 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:15:33.715641 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:33.715807 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:33.715957 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 15:15:33.716050 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20029,"thread_start_us":19904,"threads_started":1}
I20260926 15:15:33.716646 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:33.692581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:37747 (local address 127.0.0.1:52598)
0926 15:15:33.712467 (+ 19886us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:33.712485 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:33.712584 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:33.713724 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:33.713730 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:33.713808 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:33.714517 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:15:33.714521 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:33.714549 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:15:33.714626 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:33.715762 (+ 1136us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:33.715775 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:33.715782 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 15:15:33.715973 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 15:15:33.716127 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19758,"mutex_wait_us":48,"thread_start_us":15207,"threads_started":1}
I20260926 15:15:33.716871 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:33.692378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:38111 (local address 127.0.0.1:44186)
0926 15:15:33.709096 (+ 16718us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:33.709130 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:33.709259 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:33.714609 (+ 5350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:33.714615 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:33.714635 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:33.715331 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:15:33.715336 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:33.715351 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:15:33.715391 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:33.715800 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:33.715805 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:33.715809 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:15:33.716011 (+ 202us) client_negotiation.cc:240] Negotiation successful
0926 15:15:33.716083 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16481,"thread_start_us":21113,"threads_started":1}
I20260926 15:15:33.717176 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:33.692984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:45289 (local address 127.0.0.1:52596)
0926 15:15:33.713959 (+ 20975us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:33.713974 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:33.714060 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:33.714968 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:33.714974 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:33.714997 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:33.715549 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:15:33.715551 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:33.715559 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:15:33.715581 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:33.716104 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:33.716109 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:33.716118 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 15:15:33.716251 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 15:15:33.716299 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20859,"thread_start_us":20726,"threads_started":1}
I20260926 15:15:33.717206 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:33.692480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44186 (local address 127.6.209.193:38111)
0926 15:15:33.713076 (+ 20596us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:33.713086 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:33.713115 (+ 29us) server_negotiation.cc:407] Connection header received
0926 15:15:33.713181 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:33.713190 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:33.713383 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:33.714293 (+ 910us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:33.715526 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:33.715534 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:33.715545 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:15:33.715566 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:33.715614 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:15:33.715631 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:33.715747 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:33.716758 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0926 15:15:33.716875 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20427,"thread_start_us":20310,"threads_started":1}
I20260926 15:15:33.717335 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:33.693509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52596 (local address 127.6.209.194:45289)
0926 15:15:33.713512 (+ 20003us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:33.713518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:33.714179 (+ 661us) server_negotiation.cc:407] Connection header received
0926 15:15:33.714243 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:33.714249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:33.714416 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:33.714536 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:33.715696 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:33.715702 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:33.715710 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:15:33.715749 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:33.715795 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:15:33.715829 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:33.715991 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:33.716894 (+ 903us) server_negotiation.cc:299] Negotiation successful
0926 15:15:33.717012 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19869,"thread_start_us":19715,"threads_started":1}
I20260926 15:15:33.720260 12007 tablet_service.cc:1467] Processing CreateTablet for tablet 63a1dc3bd9fa40a2870446eadd1aeefa (DEFAULT_TABLE table=test-table [id=3aa3d86ef3b24a3381d48db7dd79994f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:33.720523 11747 tablet_service.cc:1467] Processing CreateTablet for tablet 63a1dc3bd9fa40a2870446eadd1aeefa (DEFAULT_TABLE table=test-table [id=3aa3d86ef3b24a3381d48db7dd79994f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:33.720523 11877 tablet_service.cc:1467] Processing CreateTablet for tablet 63a1dc3bd9fa40a2870446eadd1aeefa (DEFAULT_TABLE table=test-table [id=3aa3d86ef3b24a3381d48db7dd79994f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:33.721413 12007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a1dc3bd9fa40a2870446eadd1aeefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:33.721766 11877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a1dc3bd9fa40a2870446eadd1aeefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:33.721774 11747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a1dc3bd9fa40a2870446eadd1aeefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:33.733443 12090 tablet_bootstrap.cc:492] T 63a1dc3bd9fa40a2870446eadd1aeefa P 404955450036498b95338c11b534a4d8: Bootstrap starting.
I20260926 15:15:33.735054 12091 tablet_bootstrap.cc:492] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4: Bootstrap starting.
I20260926 15:15:33.736783 12090 tablet_bootstrap.cc:654] T 63a1dc3bd9fa40a2870446eadd1aeefa P 404955450036498b95338c11b534a4d8: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:33.737419 12092 tablet_bootstrap.cc:492] T 63a1dc3bd9fa40a2870446eadd1aeefa P 90a2d39d31d6466887087f219aa03aa6: Bootstrap starting.
I20260926 15:15:33.737977 12090 log.cc:824] T 63a1dc3bd9fa40a2870446eadd1aeefa P 404955450036498b95338c11b534a4d8: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:33.738175 12091 tablet_bootstrap.cc:654] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:33.739128 12091 log.cc:824] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:33.740427 12092 tablet_bootstrap.cc:654] T 63a1dc3bd9fa40a2870446eadd1aeefa P 90a2d39d31d6466887087f219aa03aa6: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:33.740572 12090 tablet_bootstrap.cc:492] T 63a1dc3bd9fa40a2870446eadd1aeefa P 404955450036498b95338c11b534a4d8: No bootstrap required, opened a new log
I20260926 15:15:33.741031 12090 ts_tablet_manager.cc:1397] T 63a1dc3bd9fa40a2870446eadd1aeefa P 404955450036498b95338c11b534a4d8: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 15:15:33.741532 12092 log.cc:824] T 63a1dc3bd9fa40a2870446eadd1aeefa P 90a2d39d31d6466887087f219aa03aa6: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:33.741699 12091 tablet_bootstrap.cc:492] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4: No bootstrap required, opened a new log
I20260926 15:15:33.741945 12091 ts_tablet_manager.cc:1397] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 15:15:33.744004 12092 tablet_bootstrap.cc:492] T 63a1dc3bd9fa40a2870446eadd1aeefa P 90a2d39d31d6466887087f219aa03aa6: No bootstrap required, opened a new log
I20260926 15:15:33.744206 12092 ts_tablet_manager.cc:1397] T 63a1dc3bd9fa40a2870446eadd1aeefa P 90a2d39d31d6466887087f219aa03aa6: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260926 15:15:33.750119 12092 raft_consensus.cc:348] T 63a1dc3bd9fa40a2870446eadd1aeefa P 90a2d39d31d6466887087f219aa03aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4847db036a46433f8926f218729de1d4" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38111 } } peers { permanent_uuid: "404955450036498b95338c11b534a4d8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37747 } } peers { permanent_uuid: "90a2d39d31d6466887087f219aa03aa6" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 45289 } }
I20260926 15:15:33.750418 12092 raft_consensus.cc:374] T 63a1dc3bd9fa40a2870446eadd1aeefa P 90a2d39d31d6466887087f219aa03aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:33.750507 12092 raft_consensus.cc:729] T 63a1dc3bd9fa40a2870446eadd1aeefa P 90a2d39d31d6466887087f219aa03aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a2d39d31d6466887087f219aa03aa6, State: Initialized, Role: FOLLOWER
I20260926 15:15:33.751030 12092 consensus_queue.cc:260] T 63a1dc3bd9fa40a2870446eadd1aeefa P 90a2d39d31d6466887087f219aa03aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4847db036a46433f8926f218729de1d4" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38111 } } peers { permanent_uuid: "404955450036498b95338c11b534a4d8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37747 } } peers { permanent_uuid: "90a2d39d31d6466887087f219aa03aa6" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 45289 } }
I20260926 15:15:33.750963 12090 raft_consensus.cc:348] T 63a1dc3bd9fa40a2870446eadd1aeefa P 404955450036498b95338c11b534a4d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4847db036a46433f8926f218729de1d4" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38111 } } peers { permanent_uuid: "404955450036498b95338c11b534a4d8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37747 } } peers { permanent_uuid: "90a2d39d31d6466887087f219aa03aa6" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 45289 } }
I20260926 15:15:33.751340 12090 raft_consensus.cc:374] T 63a1dc3bd9fa40a2870446eadd1aeefa P 404955450036498b95338c11b534a4d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:33.751462 12090 raft_consensus.cc:729] T 63a1dc3bd9fa40a2870446eadd1aeefa P 404955450036498b95338c11b534a4d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 404955450036498b95338c11b534a4d8, State: Initialized, Role: FOLLOWER
I20260926 15:15:33.751293 12091 raft_consensus.cc:348] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4847db036a46433f8926f218729de1d4" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38111 } } peers { permanent_uuid: "404955450036498b95338c11b534a4d8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37747 } } peers { permanent_uuid: "90a2d39d31d6466887087f219aa03aa6" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 45289 } }
I20260926 15:15:33.751679 12091 raft_consensus.cc:374] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:33.751780 12091 raft_consensus.cc:729] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4847db036a46433f8926f218729de1d4, State: Initialized, Role: FOLLOWER
I20260926 15:15:33.751884 11942 heartbeater.cc:503] Master 127.6.209.254:33253 was elected leader, sending a full tablet report...
I20260926 15:15:33.752164 12090 consensus_queue.cc:260] T 63a1dc3bd9fa40a2870446eadd1aeefa P 404955450036498b95338c11b534a4d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4847db036a46433f8926f218729de1d4" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38111 } } peers { permanent_uuid: "404955450036498b95338c11b534a4d8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37747 } } peers { permanent_uuid: "90a2d39d31d6466887087f219aa03aa6" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 45289 } }
I20260926 15:15:33.752255 12091 consensus_queue.cc:260] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4847db036a46433f8926f218729de1d4" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38111 } } peers { permanent_uuid: "404955450036498b95338c11b534a4d8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37747 } } peers { permanent_uuid: "90a2d39d31d6466887087f219aa03aa6" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 45289 } }
I20260926 15:15:33.752442 12092 ts_tablet_manager.cc:1428] T 63a1dc3bd9fa40a2870446eadd1aeefa P 90a2d39d31d6466887087f219aa03aa6: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260926 15:15:33.754091 12072 heartbeater.cc:503] Master 127.6.209.254:33253 was elected leader, sending a full tablet report...
I20260926 15:15:33.754618 12091 ts_tablet_manager.cc:1428] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4: Time spent starting tablet: real 0.013s user 0.006s sys 0.006s
I20260926 15:15:33.755064 11812 heartbeater.cc:503] Master 127.6.209.254:33253 was elected leader, sending a full tablet report...
I20260926 15:15:33.755252 12090 ts_tablet_manager.cc:1428] T 63a1dc3bd9fa40a2870446eadd1aeefa P 404955450036498b95338c11b534a4d8: Time spent starting tablet: real 0.014s user 0.006s sys 0.006s
I20260926 15:15:33.759267 12098 raft_consensus.cc:482] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:15:33.759483 12098 raft_consensus.cc:504] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4847db036a46433f8926f218729de1d4" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38111 } } peers { permanent_uuid: "404955450036498b95338c11b534a4d8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37747 } } peers { permanent_uuid: "90a2d39d31d6466887087f219aa03aa6" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 45289 } }
I20260926 15:15:33.760604 12098 leader_election.cc:288] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 404955450036498b95338c11b534a4d8 (127.6.209.195:37747), 90a2d39d31d6466887087f219aa03aa6 (127.6.209.194:45289)
W20260926 15:15:33.761132 11943 tablet.cc:2369] T 63a1dc3bd9fa40a2870446eadd1aeefa P 90a2d39d31d6466887087f219aa03aa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:15:33.771471 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:33.761140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:49139 (local address 127.6.209.195:37747)
0926 15:15:33.761777 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:33.761781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:33.769476 (+ 7695us) server_negotiation.cc:407] Connection header received
0926 15:15:33.769652 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:33.769658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:33.769705 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:33.769766 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:33.770901 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:33.770905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:33.770907 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:15:33.770920 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:33.770944 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:15:33.770954 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:33.771061 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:33.771175 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 15:15:33.771230 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559}
I20260926 15:15:33.771526 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:33.761186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:48983 (local address 127.6.209.194:45289)
0926 15:15:33.761342 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:33.761348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:33.769643 (+ 8295us) server_negotiation.cc:407] Connection header received
0926 15:15:33.769814 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:33.769818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:33.769877 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:33.769973 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:33.770942 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:33.770951 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:33.770954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:15:33.770983 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:33.771006 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:15:33.771034 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:33.771143 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:33.771255 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 15:15:33.771301 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 15:15:33.771667 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:33.760791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:37747 (local address 127.6.209.193:49139)
0926 15:15:33.769379 (+ 8588us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:33.769408 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:33.769517 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:33.769947 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:33.769954 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:33.769976 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:33.770735 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:15:33.770738 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:33.770747 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:15:33.770768 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:33.771058 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:33.771065 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:33.771072 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 15:15:33.771205 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 15:15:33.771253 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8395,"thread_start_us":8248,"threads_started":1}
I20260926 15:15:33.771771 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:33.760826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:45289 (local address 127.6.209.193:48983)
0926 15:15:33.769576 (+ 8750us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:33.769591 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:33.769693 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:33.769999 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:33.770005 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:33.770024 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:33.770687 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:15:33.770703 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:33.770720 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:15:33.770760 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:33.771151 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:33.771156 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:33.771160 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:15:33.771295 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 15:15:33.771341 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8655,"thread_start_us":8489,"threads_started":1}
I20260926 15:15:33.772488 12027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a1dc3bd9fa40a2870446eadd1aeefa" candidate_uuid: "4847db036a46433f8926f218729de1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "404955450036498b95338c11b534a4d8" is_pre_election: true
I20260926 15:15:33.772621 11897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a1dc3bd9fa40a2870446eadd1aeefa" candidate_uuid: "4847db036a46433f8926f218729de1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a2d39d31d6466887087f219aa03aa6" is_pre_election: true
I20260926 15:15:33.772948 11897 raft_consensus.cc:2454] T 63a1dc3bd9fa40a2870446eadd1aeefa P 90a2d39d31d6466887087f219aa03aa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4847db036a46433f8926f218729de1d4 in term 0.
I20260926 15:15:33.772950 12027 raft_consensus.cc:2454] T 63a1dc3bd9fa40a2870446eadd1aeefa P 404955450036498b95338c11b534a4d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4847db036a46433f8926f218729de1d4 in term 0.
I20260926 15:15:33.773437 11703 leader_election.cc:302] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4847db036a46433f8926f218729de1d4, 90a2d39d31d6466887087f219aa03aa6; no voters:
I20260926 15:15:33.773737 12098 raft_consensus.cc:2781] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:15:33.773850 12098 raft_consensus.cc:482] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:15:33.773959 12098 raft_consensus.cc:3037] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:33.775724 12098 raft_consensus.cc:504] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4847db036a46433f8926f218729de1d4" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38111 } } peers { permanent_uuid: "404955450036498b95338c11b534a4d8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37747 } } peers { permanent_uuid: "90a2d39d31d6466887087f219aa03aa6" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 45289 } }
I20260926 15:15:33.776221 12098 leader_election.cc:288] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4 [CANDIDATE]: Term 1 election: Requested vote from peers 404955450036498b95338c11b534a4d8 (127.6.209.195:37747), 90a2d39d31d6466887087f219aa03aa6 (127.6.209.194:45289)
I20260926 15:15:33.776578 12027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a1dc3bd9fa40a2870446eadd1aeefa" candidate_uuid: "4847db036a46433f8926f218729de1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "404955450036498b95338c11b534a4d8"
I20260926 15:15:33.776767 12027 raft_consensus.cc:3037] T 63a1dc3bd9fa40a2870446eadd1aeefa P 404955450036498b95338c11b534a4d8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:33.777027 11897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a1dc3bd9fa40a2870446eadd1aeefa" candidate_uuid: "4847db036a46433f8926f218729de1d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a2d39d31d6466887087f219aa03aa6"
I20260926 15:15:33.777180 11897 raft_consensus.cc:3037] T 63a1dc3bd9fa40a2870446eadd1aeefa P 90a2d39d31d6466887087f219aa03aa6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:33.778712 12027 raft_consensus.cc:2454] T 63a1dc3bd9fa40a2870446eadd1aeefa P 404955450036498b95338c11b534a4d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4847db036a46433f8926f218729de1d4 in term 1.
I20260926 15:15:33.778997 11897 raft_consensus.cc:2454] T 63a1dc3bd9fa40a2870446eadd1aeefa P 90a2d39d31d6466887087f219aa03aa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4847db036a46433f8926f218729de1d4 in term 1.
I20260926 15:15:33.779054 11701 leader_election.cc:302] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 404955450036498b95338c11b534a4d8, 4847db036a46433f8926f218729de1d4; no voters:
I20260926 15:15:33.779276 12098 raft_consensus.cc:2781] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:33.779603 12098 raft_consensus.cc:686] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4 [term 1 LEADER]: Becoming Leader. State: Replica: 4847db036a46433f8926f218729de1d4, State: Running, Role: LEADER
I20260926 15:15:33.779973 12098 consensus_queue.cc:237] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4847db036a46433f8926f218729de1d4" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38111 } } peers { permanent_uuid: "404955450036498b95338c11b534a4d8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37747 } } peers { permanent_uuid: "90a2d39d31d6466887087f219aa03aa6" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 45289 } }
I20260926 15:15:33.783762 11625 catalog_manager.cc:5495] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4847db036a46433f8926f218729de1d4 (127.6.209.193). New cstate: current_term: 1 leader_uuid: "4847db036a46433f8926f218729de1d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4847db036a46433f8926f218729de1d4" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "404955450036498b95338c11b534a4d8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a2d39d31d6466887087f219aa03aa6" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 45289 } health_report { overall_health: UNKNOWN } } }
I20260926 15:15:33.795279 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:33.793701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44188 (local address 127.6.209.193:38111)
0926 15:15:33.793811 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:33.793814 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:33.793995 (+ 181us) server_negotiation.cc:407] Connection header received
0926 15:15:33.794113 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:33.794116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:33.794167 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:33.794233 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:33.794922 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:33.794925 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:33.794927 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:15:33.794936 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:33.794956 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:15:33.794964 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:33.795033 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:33.795088 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 15:15:33.795120 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 15:15:33.802184 11897 raft_consensus.cc:1261] T 63a1dc3bd9fa40a2870446eadd1aeefa P 90a2d39d31d6466887087f219aa03aa6 [term 1 FOLLOWER]: Refusing update from remote peer 4847db036a46433f8926f218729de1d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:33.802238 12027 raft_consensus.cc:1261] T 63a1dc3bd9fa40a2870446eadd1aeefa P 404955450036498b95338c11b534a4d8 [term 1 FOLLOWER]: Refusing update from remote peer 4847db036a46433f8926f218729de1d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:33.802958 12101 consensus_queue.cc:1035] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "404955450036498b95338c11b534a4d8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:33.803236 12098 consensus_queue.cc:1035] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a2d39d31d6466887087f219aa03aa6" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 45289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:33.812850 12104 mvcc.cc:205] Tried to move back new op lower bound from 7333624765645705216 to 7333624765565685760. Current Snapshot: MvccSnapshot[applied={T|T < 7333624765645705216}]
I20260926 15:15:33.825608 11626 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:33.825884 11626 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:33.829285 11626 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1814e5d2cfb3432782bbe5f55e832dde: Sending DeleteTablet for 3 replicas of tablet 63a1dc3bd9fa40a2870446eadd1aeefa
I20260926 15:15:33.830180 11747 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a1dc3bd9fa40a2870446eadd1aeefa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:33 UTC) from {username='slave'} at 127.0.0.1:44186
I20260926 15:15:33.830214 12007 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a1dc3bd9fa40a2870446eadd1aeefa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:33 UTC) from {username='slave'} at 127.0.0.1:52598
I20260926 15:15:33.830262 11877 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a1dc3bd9fa40a2870446eadd1aeefa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:33 UTC) from {username='slave'} at 127.0.0.1:52596
I20260926 15:15:33.833230 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 11685
I20260926 15:15:33.840914 12115 tablet_replica.cc:330] T 63a1dc3bd9fa40a2870446eadd1aeefa P 404955450036498b95338c11b534a4d8: stopping tablet replica
I20260926 15:15:33.841399 12115 raft_consensus.cc:2229] T 63a1dc3bd9fa40a2870446eadd1aeefa P 404955450036498b95338c11b534a4d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:33.841849 12115 raft_consensus.cc:2258] T 63a1dc3bd9fa40a2870446eadd1aeefa P 404955450036498b95338c11b534a4d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:33.842696 12113 tablet_replica.cc:330] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4: stopping tablet replica
I20260926 15:15:33.843776 12115 ts_tablet_manager.cc:1905] T 63a1dc3bd9fa40a2870446eadd1aeefa P 404955450036498b95338c11b534a4d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:33.844218 12114 tablet_replica.cc:330] T 63a1dc3bd9fa40a2870446eadd1aeefa P 90a2d39d31d6466887087f219aa03aa6: stopping tablet replica
I20260926 15:15:33.844543 12114 raft_consensus.cc:2229] T 63a1dc3bd9fa40a2870446eadd1aeefa P 90a2d39d31d6466887087f219aa03aa6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:33.844928 12113 raft_consensus.cc:2229] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:15:33.845430 12113 raft_consensus.cc:2258] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:33.845993 12114 raft_consensus.cc:2258] T 63a1dc3bd9fa40a2870446eadd1aeefa P 90a2d39d31d6466887087f219aa03aa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:33.849434 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:33.834639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44190 (local address 127.6.209.193:38111)
0926 15:15:33.834839 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:33.834846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:33.847666 (+ 12820us) server_negotiation.cc:407] Connection header received
0926 15:15:33.847850 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:33.847855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:33.847905 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:33.848000 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:33.848736 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:33.848741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:33.848745 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:15:33.848759 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:33.848791 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:15:33.848800 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:33.848932 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:33.849127 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 15:15:33.849179 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 15:15:33.850446 11807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:33.850473 12114 ts_tablet_manager.cc:1905] T 63a1dc3bd9fa40a2870446eadd1aeefa P 90a2d39d31d6466887087f219aa03aa6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:33.852856 12115 ts_tablet_manager.cc:1918] T 63a1dc3bd9fa40a2870446eadd1aeefa P 404955450036498b95338c11b534a4d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:15:33.853068 12115 log.cc:1196] T 63a1dc3bd9fa40a2870446eadd1aeefa P 404955450036498b95338c11b534a4d8: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/63a1dc3bd9fa40a2870446eadd1aeefa
I20260926 15:15:33.853600 12115 ts_tablet_manager.cc:1939] T 63a1dc3bd9fa40a2870446eadd1aeefa P 404955450036498b95338c11b534a4d8: Deleting consensus metadata
I20260926 15:15:33.855093 11611 catalog_manager.cc:4841] TS 404955450036498b95338c11b534a4d8 (127.6.209.195:37747): tablet 63a1dc3bd9fa40a2870446eadd1aeefa (table test-table [id=3aa3d86ef3b24a3381d48db7dd79994f]) successfully deleted
I20260926 15:15:33.857023 12114 ts_tablet_manager.cc:1918] T 63a1dc3bd9fa40a2870446eadd1aeefa P 90a2d39d31d6466887087f219aa03aa6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:15:33.857161 12114 log.cc:1196] T 63a1dc3bd9fa40a2870446eadd1aeefa P 90a2d39d31d6466887087f219aa03aa6: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/63a1dc3bd9fa40a2870446eadd1aeefa
I20260926 15:15:33.857589 12114 ts_tablet_manager.cc:1939] T 63a1dc3bd9fa40a2870446eadd1aeefa P 90a2d39d31d6466887087f219aa03aa6: Deleting consensus metadata
I20260926 15:15:33.858742 11613 catalog_manager.cc:4841] TS 90a2d39d31d6466887087f219aa03aa6 (127.6.209.194:45289): tablet 63a1dc3bd9fa40a2870446eadd1aeefa (table test-table [id=3aa3d86ef3b24a3381d48db7dd79994f]) successfully deleted
I20260926 15:15:33.935683 12113 ts_tablet_manager.cc:1905] T 63a1dc3bd9fa40a2870446eadd1aeefa P 4847db036a46433f8926f218729de1d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:33.941545 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 11685
W20260926 15:15:33.958688 11610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.209.193:38111 (error 108)
I20260926 15:15:33.958895 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 11815
I20260926 15:15:33.960292 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:33.959959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:38111 (local address 127.0.0.1:44192)
0926 15:15:33.960086 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:33.960181 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:38111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 15:15:33.960536 11610 catalog_manager.cc:4584] TS 4847db036a46433f8926f218729de1d4 (127.6.209.193:38111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63a1dc3bd9fa40a2870446eadd1aeefa: Network error: Client connection negotiation failed: client connection to 127.6.209.193:38111: connect: Connection refused (error 111)
I20260926 15:15:33.961711 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:33.959616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52608 (local address 127.6.209.194:45289)
0926 15:15:33.959806 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:33.959813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:33.959835 (+ 22us) server_negotiation.cc:407] Connection header received
0926 15:15:33.960001 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:33.960006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:33.960109 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:33.960214 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:33.961125 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:33.961130 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:33.961135 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:15:33.961153 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:33.961188 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:15:33.961200 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:33.961326 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:33.961408 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 15:15:33.961464 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 15:15:33.962251 11937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:34.013973 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:34.013588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:38111 (local address 127.0.0.1:44196)
0926 15:15:34.013745 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:34.013834 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:38111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 15:15:34.014719 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:34.014450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:38111 (local address 127.0.0.1:44200)
0926 15:15:34.014572 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:34.014631 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:38111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 15:15:34.041673 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 11815
I20260926 15:15:34.055740 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 11945
I20260926 15:15:34.058073 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:34.056405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52610 (local address 127.6.209.195:37747)
0926 15:15:34.056571 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:34.056576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:34.056593 (+ 17us) server_negotiation.cc:407] Connection header received
0926 15:15:34.056694 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:34.056698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:34.056778 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:34.056909 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:34.057580 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:34.057584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:34.057586 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:15:34.057598 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:34.057622 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:15:34.057632 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:34.057718 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:34.057771 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 15:15:34.057807 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 15:15:34.058601 12067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:34.074777 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:34.074353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:38111 (local address 127.0.0.1:44204)
0926 15:15:34.074539 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:34.074627 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:38111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 15:15:34.075598 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:34.075309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:38111 (local address 127.0.0.1:44214)
0926 15:15:34.075445 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:34.075511 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:38111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 15:15:34.137507 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 11945
I20260926 15:15:34.153214 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 11595
I20260926 15:15:34.154201 11656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:34.248342 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:34.247910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:38111 (local address 127.0.0.1:44226)
0926 15:15:34.248093 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:34.248204 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:38111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 15:15:34.248409 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 11595
2026-09-26T15:15:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2268 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 15:15:34.283529 6983 mini_oidc.cc:152] Starting JWKS server
I20260926 15:15:34.285856 6983 webserver.cc:460] Webserver started at https://127.0.0.1:43623/ using document root <none> and password file <none>
I20260926 15:15:34.286175 6983 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 15:15:34.286757 6983 webserver.cc:460] Webserver started at http://127.0.0.1:42311/ using document root <none> and password file <none>
2026-09-26T15:15:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:15:34Z Disabled control of system clock
I20260926 15:15:34.309144 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:46367
--webserver_interface=127.6.209.254
--webserver_port=0
--builtin_ntp_servers=127.6.209.212:32945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:46367
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43623/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435694426955-6983-0/testchainca.cert with env {}
W20260926 15:15:34.596448 12127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:34.596714 12127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:34.596841 12127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:34.603618 12127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:15:34.603704 12127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:34.603746 12127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:34.603794 12127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:15:34.603834 12127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 15:15:34.603892 12127 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 15:15:34.603932 12127 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43623/jwks.json
I20260926 15:15:34.612608 12127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:32945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:46367
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:46367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43623/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435694426955-6983-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:34.613767 12127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:34.614880 12127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:34.629244 12132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:34.629315 12133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:34.629273 12135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:34.629879 12127 server_base.cc:1034] running on GCE node
I20260926 15:15:34.630576 12127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:34.631682 12127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:34.632927 12127 hybrid_clock.cc:648] HybridClock initialized: now 1790435734632895 us; error 29 us; skew 500 ppm
I20260926 15:15:34.633245 12127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:15:34.635267 12127 webserver.cc:460] Webserver started at http://127.6.209.254:38619/ using document root <none> and password file <none>
I20260926 15:15:34.635845 12127 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:34.635979 12127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:34.636262 12127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:34.638993 12127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "e285f77f3f3a4c7088489cf4b65e3be3"
format_stamp: "Formatted at 2026-09-26 15:15:34 on dist-test-slave-j6q7"
server_key: "4ec9415e9ad4cc8f87bd7957903a1e11"
server_key_iv: "ecf2655f970331660909e7a6da7e1ba4"
server_key_version: "encryptionkey@0"
I20260926 15:15:34.639990 12127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "e285f77f3f3a4c7088489cf4b65e3be3"
format_stamp: "Formatted at 2026-09-26 15:15:34 on dist-test-slave-j6q7"
server_key: "4ec9415e9ad4cc8f87bd7957903a1e11"
server_key_iv: "ecf2655f970331660909e7a6da7e1ba4"
server_key_version: "encryptionkey@0"
I20260926 15:15:34.646260 12127 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260926 15:15:34.649452 12141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:34.650696 12127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:15:34.650826 12127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "e285f77f3f3a4c7088489cf4b65e3be3"
format_stamp: "Formatted at 2026-09-26 15:15:34 on dist-test-slave-j6q7"
server_key: "4ec9415e9ad4cc8f87bd7957903a1e11"
server_key_iv: "ecf2655f970331660909e7a6da7e1ba4"
server_key_version: "encryptionkey@0"
I20260926 15:15:34.650995 12127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:34.677393 12127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:34.678069 12127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:34.678311 12127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:34.694804 12127 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:46367
I20260926 15:15:34.694823 12192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:46367 every 8 connection(s)
I20260926 15:15:34.696002 12127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:15:34.699425 12193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:34.702467 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 12127
I20260926 15:15:34.702644 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:15:34.703414 6983 external_mini_cluster.cc:1428] Setting key 64e36b74b0fee6a5ad97537dba10343b
I20260926 15:15:34.711069 12193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e285f77f3f3a4c7088489cf4b65e3be3: Bootstrap starting.
I20260926 15:15:34.714417 12193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e285f77f3f3a4c7088489cf4b65e3be3: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:34.715595 12193 log.cc:824] T 00000000000000000000000000000000 P e285f77f3f3a4c7088489cf4b65e3be3: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:34.716452 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:34.705159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46682 (local address 127.6.209.254:46367)
0926 15:15:34.705956 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:34.705966 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:34.706004 (+ 38us) server_negotiation.cc:407] Connection header received
0926 15:15:34.707585 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:34.707606 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:34.708023 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:34.708330 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:34.708943 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:34.710146 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:34.711776 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:34.712416 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:34.712544 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:34.713761 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:34.713768 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:34.713780 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:15:34.714259 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:34.714350 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:15:34.714375 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:34.714460 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:34.714968 (+ 508us) server_negotiation.cc:299] Negotiation successful
0926 15:15:34.715243 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":303,"threads_started":1}
I20260926 15:15:34.718466 12193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e285f77f3f3a4c7088489cf4b65e3be3: No bootstrap required, opened a new log
I20260926 15:15:34.720208 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:15:34.726908 12193 raft_consensus.cc:348] T 00000000000000000000000000000000 P e285f77f3f3a4c7088489cf4b65e3be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e285f77f3f3a4c7088489cf4b65e3be3" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 46367 } }
I20260926 15:15:34.727186 12193 raft_consensus.cc:374] T 00000000000000000000000000000000 P e285f77f3f3a4c7088489cf4b65e3be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:34.727258 12193 raft_consensus.cc:729] T 00000000000000000000000000000000 P e285f77f3f3a4c7088489cf4b65e3be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e285f77f3f3a4c7088489cf4b65e3be3, State: Initialized, Role: FOLLOWER
I20260926 15:15:34.727807 12193 consensus_queue.cc:260] T 00000000000000000000000000000000 P e285f77f3f3a4c7088489cf4b65e3be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e285f77f3f3a4c7088489cf4b65e3be3" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 46367 } }
I20260926 15:15:34.728037 12193 raft_consensus.cc:388] T 00000000000000000000000000000000 P e285f77f3f3a4c7088489cf4b65e3be3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:15:34.728170 12193 raft_consensus.cc:482] T 00000000000000000000000000000000 P e285f77f3f3a4c7088489cf4b65e3be3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:15:34.728345 12193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e285f77f3f3a4c7088489cf4b65e3be3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:34.730317 12193 raft_consensus.cc:504] T 00000000000000000000000000000000 P e285f77f3f3a4c7088489cf4b65e3be3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e285f77f3f3a4c7088489cf4b65e3be3" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 46367 } }
I20260926 15:15:34.730744 12193 leader_election.cc:302] T 00000000000000000000000000000000 P e285f77f3f3a4c7088489cf4b65e3be3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e285f77f3f3a4c7088489cf4b65e3be3; no voters:
I20260926 15:15:34.731207 12193 leader_election.cc:288] T 00000000000000000000000000000000 P e285f77f3f3a4c7088489cf4b65e3be3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:15:34.731365 12198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e285f77f3f3a4c7088489cf4b65e3be3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:34.732040 12198 raft_consensus.cc:686] T 00000000000000000000000000000000 P e285f77f3f3a4c7088489cf4b65e3be3 [term 1 LEADER]: Becoming Leader. State: Replica: e285f77f3f3a4c7088489cf4b65e3be3, State: Running, Role: LEADER
I20260926 15:15:34.732573 12198 consensus_queue.cc:237] T 00000000000000000000000000000000 P e285f77f3f3a4c7088489cf4b65e3be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e285f77f3f3a4c7088489cf4b65e3be3" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 46367 } }
I20260926 15:15:34.733029 12193 sys_catalog.cc:564] T 00000000000000000000000000000000 P e285f77f3f3a4c7088489cf4b65e3be3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:15:34.735213 12199 sys_catalog.cc:455] T 00000000000000000000000000000000 P e285f77f3f3a4c7088489cf4b65e3be3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e285f77f3f3a4c7088489cf4b65e3be3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e285f77f3f3a4c7088489cf4b65e3be3" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 46367 } } }
I20260926 15:15:34.735620 12199 sys_catalog.cc:458] T 00000000000000000000000000000000 P e285f77f3f3a4c7088489cf4b65e3be3 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:34.735201 12200 sys_catalog.cc:455] T 00000000000000000000000000000000 P e285f77f3f3a4c7088489cf4b65e3be3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e285f77f3f3a4c7088489cf4b65e3be3. Latest consensus state: current_term: 1 leader_uuid: "e285f77f3f3a4c7088489cf4b65e3be3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e285f77f3f3a4c7088489cf4b65e3be3" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 46367 } } }
I20260926 15:15:34.736266 12200 sys_catalog.cc:458] T 00000000000000000000000000000000 P e285f77f3f3a4c7088489cf4b65e3be3 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:34.737082 12206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:15:34.746405 12206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:15:34.754792 12206 catalog_manager.cc:1347] Generated new cluster ID: b38823cb86b34608a0bfa84a7d65d5bd
I20260926 15:15:34.754889 12206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:15:34.786269 12206 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:15:34.787428 12206 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:15:34.796379 12206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e285f77f3f3a4c7088489cf4b65e3be3: Generated new TSK 0
I20260926 15:15:34.797371 12206 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 15:15:34.830854 12223 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 15:15:34.830996 12223 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 15:15:34.832086 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:34.812551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46694 (local address 127.6.209.254:46367)
0926 15:15:34.812705 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:34.812711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:34.813150 (+ 439us) server_negotiation.cc:407] Connection header received
0926 15:15:34.813370 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:34.813375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:34.813447 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:34.813592 (+ 145us) server_negotiation.cc:226] Negotiated authn=JWT
0926 15:15:34.814019 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:34.815055 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:34.816234 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:34.816935 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:34.817053 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:34.817453 (+ 400us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 15:15:34.831508 (+ 14055us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43623/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 15:15:34.831795 (+ 287us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46694: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43623/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 15:15:34.832194 12196 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46694: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43623/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 15:15:34.833352 12222 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:15:34.812468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:46367 (local address 127.0.0.1:46694)
0926 15:15:34.813073 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:34.813087 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:34.813200 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:34.813621 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:34.813624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:34.813632 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 15:15:34.813897 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:34.813904 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:34.815151 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:34.815154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:34.816109 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:34.816119 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:34.817082 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:34.817087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:34.817209 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:34.817252 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 15:15:34.831718 (+ 14466us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43623/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 15:15:34.831923 (+ 205us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.209.254:46367: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43623/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":341,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.6.209.254:46367: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43623/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 15:15:34.834836 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 12127
I20260926 15:15:34.835745 12188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:34.926465 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 12127
2026-09-26T15:15:34Z chronyd exiting
I20260926 15:15:34.978479 6983 test_util.cc:175] -----------------------------------------------
I20260926 15:15:34.978587 6983 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435694426955-6983-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (698 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 15:15:34.982637 6983 mini_oidc.cc:152] Starting JWKS server
I20260926 15:15:34.984342 6983 webserver.cc:460] Webserver started at https://127.0.0.1:37457/ using document root <none> and password file <none>
I20260926 15:15:34.984558 6983 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 15:15:34.985193 6983 webserver.cc:460] Webserver started at http://127.0.0.1:39813/ using document root <none> and password file <none>
2026-09-26T15:15:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:15:34Z Disabled control of system clock
I20260926 15:15:35.009635 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:33627
--webserver_interface=127.6.209.254
--webserver_port=0
--builtin_ntp_servers=127.6.209.212:32853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:33627
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37457/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435694426955-6983-0/testchainca.cert with env {}
W20260926 15:15:35.304651 12230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:35.304935 12230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:35.305011 12230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:35.311404 12230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:15:35.311492 12230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:35.311523 12230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:35.311563 12230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:15:35.311592 12230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 15:15:35.311636 12230 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 15:15:35.311671 12230 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37457/jwks.json
I20260926 15:15:35.319758 12230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:32853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:33627
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:33627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37457/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435694426955-6983-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:35.320700 12230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:35.321825 12230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:35.333148 12235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:35.333226 12238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:35.333194 12236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:35.333773 12230 server_base.cc:1034] running on GCE node
I20260926 15:15:35.334455 12230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:35.335621 12230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:35.336879 12230 hybrid_clock.cc:648] HybridClock initialized: now 1790435735336796 us; error 38 us; skew 500 ppm
I20260926 15:15:35.337267 12230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:15:35.339351 12230 webserver.cc:460] Webserver started at http://127.6.209.254:43893/ using document root <none> and password file <none>
I20260926 15:15:35.339937 12230 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:35.340054 12230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:35.340339 12230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:35.343035 12230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "4e4d82f044e3458fb01d26f6ed95ae05"
format_stamp: "Formatted at 2026-09-26 15:15:35 on dist-test-slave-j6q7"
server_key: "8325ceb11e4e13eb82c49882bd30b3cc"
server_key_iv: "5a5ad6aff0493202f19f343dcd9ddf7c"
server_key_version: "encryptionkey@0"
I20260926 15:15:35.343753 12230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "4e4d82f044e3458fb01d26f6ed95ae05"
format_stamp: "Formatted at 2026-09-26 15:15:35 on dist-test-slave-j6q7"
server_key: "8325ceb11e4e13eb82c49882bd30b3cc"
server_key_iv: "5a5ad6aff0493202f19f343dcd9ddf7c"
server_key_version: "encryptionkey@0"
I20260926 15:15:35.348385 12230 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260926 15:15:35.351257 12244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:35.352421 12230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:15:35.352610 12230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "4e4d82f044e3458fb01d26f6ed95ae05"
format_stamp: "Formatted at 2026-09-26 15:15:35 on dist-test-slave-j6q7"
server_key: "8325ceb11e4e13eb82c49882bd30b3cc"
server_key_iv: "5a5ad6aff0493202f19f343dcd9ddf7c"
server_key_version: "encryptionkey@0"
I20260926 15:15:35.352875 12230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:35.406461 12230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:35.407208 12230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:35.407492 12230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:35.423069 12230 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:33627
I20260926 15:15:35.423086 12295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:33627 every 8 connection(s)
I20260926 15:15:35.424201 12230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:15:35.427310 12296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:35.433944 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 12230
I20260926 15:15:35.434139 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:15:35.434650 6983 external_mini_cluster.cc:1428] Setting key a90fe49b346439c1a8eeb2a8971a99e6
I20260926 15:15:35.436197 12296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e4d82f044e3458fb01d26f6ed95ae05: Bootstrap starting.
I20260926 15:15:35.438939 12296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e4d82f044e3458fb01d26f6ed95ae05: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:35.439996 12296 log.cc:824] T 00000000000000000000000000000000 P 4e4d82f044e3458fb01d26f6ed95ae05: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:35.442742 12296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e4d82f044e3458fb01d26f6ed95ae05: No bootstrap required, opened a new log
I20260926 15:15:35.446518 12299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:35.436225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50568 (local address 127.6.209.254:33627)
0926 15:15:35.437158 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:35.437168 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:35.437214 (+ 46us) server_negotiation.cc:407] Connection header received
0926 15:15:35.438631 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:35.438649 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:35.439019 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:35.439335 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:35.439798 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:35.441357 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:35.443010 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:35.443537 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:35.443638 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:35.444580 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:35.444589 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:35.444600 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:15:35.445085 (+ 485us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:35.445149 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:15:35.445165 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:35.445244 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:35.445418 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 15:15:35.445715 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":324,"threads_started":1}
I20260926 15:15:35.449922 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:15:35.450533 12296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e4d82f044e3458fb01d26f6ed95ae05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e4d82f044e3458fb01d26f6ed95ae05" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 33627 } }
I20260926 15:15:35.450795 12296 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e4d82f044e3458fb01d26f6ed95ae05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:35.450891 12296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e4d82f044e3458fb01d26f6ed95ae05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4d82f044e3458fb01d26f6ed95ae05, State: Initialized, Role: FOLLOWER
I20260926 15:15:35.451376 12296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e4d82f044e3458fb01d26f6ed95ae05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e4d82f044e3458fb01d26f6ed95ae05" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 33627 } }
I20260926 15:15:35.451560 12296 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e4d82f044e3458fb01d26f6ed95ae05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:15:35.451668 12296 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e4d82f044e3458fb01d26f6ed95ae05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:15:35.451819 12296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e4d82f044e3458fb01d26f6ed95ae05 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:35.453920 12296 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e4d82f044e3458fb01d26f6ed95ae05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e4d82f044e3458fb01d26f6ed95ae05" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 33627 } }
I20260926 15:15:35.454365 12296 leader_election.cc:302] T 00000000000000000000000000000000 P 4e4d82f044e3458fb01d26f6ed95ae05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e4d82f044e3458fb01d26f6ed95ae05; no voters:
I20260926 15:15:35.454779 12296 leader_election.cc:288] T 00000000000000000000000000000000 P 4e4d82f044e3458fb01d26f6ed95ae05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:15:35.454854 12301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e4d82f044e3458fb01d26f6ed95ae05 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:35.455447 12301 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e4d82f044e3458fb01d26f6ed95ae05 [term 1 LEADER]: Becoming Leader. State: Replica: 4e4d82f044e3458fb01d26f6ed95ae05, State: Running, Role: LEADER
I20260926 15:15:35.455909 12301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e4d82f044e3458fb01d26f6ed95ae05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e4d82f044e3458fb01d26f6ed95ae05" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 33627 } }
I20260926 15:15:35.456568 12296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e4d82f044e3458fb01d26f6ed95ae05 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:15:35.458477 12303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e4d82f044e3458fb01d26f6ed95ae05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e4d82f044e3458fb01d26f6ed95ae05. Latest consensus state: current_term: 1 leader_uuid: "4e4d82f044e3458fb01d26f6ed95ae05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e4d82f044e3458fb01d26f6ed95ae05" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 33627 } } }
I20260926 15:15:35.458513 12302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e4d82f044e3458fb01d26f6ed95ae05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e4d82f044e3458fb01d26f6ed95ae05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e4d82f044e3458fb01d26f6ed95ae05" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 33627 } } }
I20260926 15:15:35.458890 12303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e4d82f044e3458fb01d26f6ed95ae05 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:35.458956 12302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e4d82f044e3458fb01d26f6ed95ae05 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:35.459707 12309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:15:35.465495 12309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:15:35.476190 12309 catalog_manager.cc:1347] Generated new cluster ID: 0ecef37f112a49f0ae5df3aca9c55922
I20260926 15:15:35.476297 12309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:15:35.491329 12309 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:15:35.492717 12309 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:15:35.511978 12309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e4d82f044e3458fb01d26f6ed95ae05: Generated new TSK 0
I20260926 15:15:35.512838 12309 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 15:15:35.543105 12326 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 15:15:35.543200 12326 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260926 15:15:35.544181 12325 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:15:35.526428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:33627 (local address 127.0.0.1:50580)
0926 15:15:35.527055 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:35.527073 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:35.527242 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:35.527657 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:35.527660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:35.527668 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 15:15:35.527874 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:35.527881 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:35.529129 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:35.529132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:35.530096 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:35.530105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:35.531029 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:35.531044 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:35.531200 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:35.531245 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 15:15:35.543804 (+ 12559us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37457/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 15:15:35.543949 (+ 145us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.209.254:33627: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37457/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":326,"threads_started":1}
I20260926 15:15:35.544178 12299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:35.526442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50580 (local address 127.6.209.254:33627)
0926 15:15:35.526628 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:35.526638 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:35.527154 (+ 516us) server_negotiation.cc:407] Connection header received
0926 15:15:35.527421 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:35.527426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:35.527506 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:35.527628 (+ 122us) server_negotiation.cc:226] Negotiated authn=JWT
0926 15:15:35.528019 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:35.529026 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:35.530226 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:35.530892 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:35.531019 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:35.531457 (+ 438us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 15:15:35.543650 (+ 12193us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37457/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 15:15:35.543942 (+ 292us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50580: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37457/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 15:15:35.544534 12299 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50580: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37457/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260926 15:15:35.545007 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 12230
I20260926 15:15:35.545935 12291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:35.631198 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 12230
2026-09-26T15:15:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (706 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 15:15:35.688107 6983 mini_oidc.cc:152] Starting JWKS server
I20260926 15:15:35.689903 6983 webserver.cc:460] Webserver started at https://127.0.0.1:42799/ using document root <none> and password file <none>
I20260926 15:15:35.690124 6983 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 15:15:35.690713 6983 webserver.cc:460] Webserver started at http://127.0.0.1:35339/ using document root <none> and password file <none>
2026-09-26T15:15:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:15:35Z Disabled control of system clock
I20260926 15:15:35.713922 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:42901
--webserver_interface=127.6.209.254
--webserver_port=0
--builtin_ntp_servers=127.6.209.212:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:42901
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42799/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 15:15:35.994757 12333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:35.994977 12333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:35.995051 12333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:36.001628 12333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:15:36.001710 12333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:36.001746 12333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:36.001801 12333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:15:36.001840 12333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 15:15:36.001896 12333 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 15:15:36.001935 12333 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42799/jwks.json
I20260926 15:15:36.010027 12333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:42901
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:42901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42799/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:36.011021 12333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:36.012451 12333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:36.022284 12339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:36.022253 12341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:36.022531 12338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:36.023119 12333 server_base.cc:1034] running on GCE node
I20260926 15:15:36.023741 12333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:36.024920 12333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:36.026137 12333 hybrid_clock.cc:648] HybridClock initialized: now 1790435736026093 us; error 41 us; skew 500 ppm
I20260926 15:15:36.026506 12333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:15:36.028535 12333 webserver.cc:460] Webserver started at http://127.6.209.254:39679/ using document root <none> and password file <none>
I20260926 15:15:36.029163 12333 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:36.029268 12333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:36.029541 12333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:36.032020 12333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "b94a4bdb748c4f1c96862a4df83ff2c8"
format_stamp: "Formatted at 2026-09-26 15:15:36 on dist-test-slave-j6q7"
server_key: "348b721175113492829a6f36902d3865"
server_key_iv: "b61410c474ccdc3330d010c0ef019029"
server_key_version: "encryptionkey@0"
I20260926 15:15:36.032687 12333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "b94a4bdb748c4f1c96862a4df83ff2c8"
format_stamp: "Formatted at 2026-09-26 15:15:36 on dist-test-slave-j6q7"
server_key: "348b721175113492829a6f36902d3865"
server_key_iv: "b61410c474ccdc3330d010c0ef019029"
server_key_version: "encryptionkey@0"
I20260926 15:15:36.037139 12333 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:15:36.039924 12347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:36.041074 12333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:15:36.041225 12333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "b94a4bdb748c4f1c96862a4df83ff2c8"
format_stamp: "Formatted at 2026-09-26 15:15:36 on dist-test-slave-j6q7"
server_key: "348b721175113492829a6f36902d3865"
server_key_iv: "b61410c474ccdc3330d010c0ef019029"
server_key_version: "encryptionkey@0"
I20260926 15:15:36.041417 12333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:36.056953 12333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:36.057660 12333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:36.057930 12333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:36.073174 12333 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:42901
I20260926 15:15:36.073200 12398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:42901 every 8 connection(s)
I20260926 15:15:36.074230 12333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:15:36.077286 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 12333
I20260926 15:15:36.077425 12399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:36.077446 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:15:36.078042 6983 external_mini_cluster.cc:1428] Setting key 1ea1583b5f3b1eb8a8b0451cba07124f
I20260926 15:15:36.088577 12399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b94a4bdb748c4f1c96862a4df83ff2c8: Bootstrap starting.
I20260926 15:15:36.089847 12402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:36.079585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41130 (local address 127.6.209.254:42901)
0926 15:15:36.080455 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:36.080464 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:36.080498 (+ 34us) server_negotiation.cc:407] Connection header received
0926 15:15:36.082369 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:36.082388 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:36.082784 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:36.083066 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:36.083602 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:36.084598 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:36.086226 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:36.086853 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:36.086969 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:36.087997 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:36.088003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:36.088013 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:15:36.088433 (+ 420us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:36.088495 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:15:36.088511 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:36.088585 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:36.088801 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 15:15:36.089099 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":334,"threads_started":1}
I20260926 15:15:36.091326 12399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b94a4bdb748c4f1c96862a4df83ff2c8: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:36.092396 12399 log.cc:824] T 00000000000000000000000000000000 P b94a4bdb748c4f1c96862a4df83ff2c8: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:36.093026 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:15:36.094594 12399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b94a4bdb748c4f1c96862a4df83ff2c8: No bootstrap required, opened a new log
I20260926 15:15:36.101680 12399 raft_consensus.cc:348] T 00000000000000000000000000000000 P b94a4bdb748c4f1c96862a4df83ff2c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94a4bdb748c4f1c96862a4df83ff2c8" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 42901 } }
I20260926 15:15:36.102002 12399 raft_consensus.cc:374] T 00000000000000000000000000000000 P b94a4bdb748c4f1c96862a4df83ff2c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:36.102100 12399 raft_consensus.cc:729] T 00000000000000000000000000000000 P b94a4bdb748c4f1c96862a4df83ff2c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94a4bdb748c4f1c96862a4df83ff2c8, State: Initialized, Role: FOLLOWER
I20260926 15:15:36.102593 12399 consensus_queue.cc:260] T 00000000000000000000000000000000 P b94a4bdb748c4f1c96862a4df83ff2c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94a4bdb748c4f1c96862a4df83ff2c8" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 42901 } }
I20260926 15:15:36.102803 12399 raft_consensus.cc:388] T 00000000000000000000000000000000 P b94a4bdb748c4f1c96862a4df83ff2c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:15:36.102943 12399 raft_consensus.cc:482] T 00000000000000000000000000000000 P b94a4bdb748c4f1c96862a4df83ff2c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:15:36.103091 12399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b94a4bdb748c4f1c96862a4df83ff2c8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:36.105049 12399 raft_consensus.cc:504] T 00000000000000000000000000000000 P b94a4bdb748c4f1c96862a4df83ff2c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94a4bdb748c4f1c96862a4df83ff2c8" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 42901 } }
I20260926 15:15:36.105463 12399 leader_election.cc:302] T 00000000000000000000000000000000 P b94a4bdb748c4f1c96862a4df83ff2c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b94a4bdb748c4f1c96862a4df83ff2c8; no voters:
I20260926 15:15:36.105963 12399 leader_election.cc:288] T 00000000000000000000000000000000 P b94a4bdb748c4f1c96862a4df83ff2c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:15:36.106132 12404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b94a4bdb748c4f1c96862a4df83ff2c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:36.106817 12404 raft_consensus.cc:686] T 00000000000000000000000000000000 P b94a4bdb748c4f1c96862a4df83ff2c8 [term 1 LEADER]: Becoming Leader. State: Replica: b94a4bdb748c4f1c96862a4df83ff2c8, State: Running, Role: LEADER
I20260926 15:15:36.107376 12404 consensus_queue.cc:237] T 00000000000000000000000000000000 P b94a4bdb748c4f1c96862a4df83ff2c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94a4bdb748c4f1c96862a4df83ff2c8" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 42901 } }
I20260926 15:15:36.107801 12399 sys_catalog.cc:564] T 00000000000000000000000000000000 P b94a4bdb748c4f1c96862a4df83ff2c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:15:36.109880 12405 sys_catalog.cc:455] T 00000000000000000000000000000000 P b94a4bdb748c4f1c96862a4df83ff2c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b94a4bdb748c4f1c96862a4df83ff2c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94a4bdb748c4f1c96862a4df83ff2c8" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 42901 } } }
I20260926 15:15:36.109872 12406 sys_catalog.cc:455] T 00000000000000000000000000000000 P b94a4bdb748c4f1c96862a4df83ff2c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b94a4bdb748c4f1c96862a4df83ff2c8. Latest consensus state: current_term: 1 leader_uuid: "b94a4bdb748c4f1c96862a4df83ff2c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94a4bdb748c4f1c96862a4df83ff2c8" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 42901 } } }
I20260926 15:15:36.110237 12405 sys_catalog.cc:458] T 00000000000000000000000000000000 P b94a4bdb748c4f1c96862a4df83ff2c8 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:36.110278 12406 sys_catalog.cc:458] T 00000000000000000000000000000000 P b94a4bdb748c4f1c96862a4df83ff2c8 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:36.111819 12410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:15:36.116343 12410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:15:36.125458 12410 catalog_manager.cc:1347] Generated new cluster ID: 6f8cf6968c6f4a7aa213b10984bb4000
I20260926 15:15:36.125553 12410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:15:36.146534 12410 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:15:36.147737 12410 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:15:36.157786 12410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b94a4bdb748c4f1c96862a4df83ff2c8: Generated new TSK 0
I20260926 15:15:36.158507 12410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:15:36.176847 12402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:36.169637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41140 (local address 127.6.209.254:42901)
0926 15:15:36.169788 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:36.169793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:36.170140 (+ 347us) server_negotiation.cc:407] Connection header received
0926 15:15:36.170353 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:36.170359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:36.170457 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:36.170604 (+ 147us) server_negotiation.cc:226] Negotiated authn=JWT
0926 15:15:36.170943 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:36.171933 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:36.173154 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:36.173785 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:36.173893 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:36.174274 (+ 381us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 15:15:36.176234 (+ 1960us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42799/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 15:15:36.176537 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41140: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42799/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 15:15:36.176954 12402 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41140: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42799/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 15:15:36.178402 12428 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:15:36.169514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:42901 (local address 127.0.0.1:41140)
0926 15:15:36.170039 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:36.170052 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:36.170163 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:36.170604 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:36.170607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:36.170615 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 15:15:36.170824 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:36.170831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:36.172044 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:36.172047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:36.173042 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:36.173051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:36.173922 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:36.173925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:36.174023 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:36.174047 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 15:15:36.176426 (+ 2379us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42799/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 15:15:36.178084 (+ 1658us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.209.254:42901: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42799/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":1414,"thread_start_us":285,"threads_started":1}
I20260926 15:15:36.179358 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 12333
I20260926 15:15:36.180253 12394 generic_service.cc:196] Checking for leaks (request via RPC)
E20260926 15:15:36.187606 12429 webserver.cc:533] Webserver: sslize failed:
E20260926 15:15:36.187683 12429 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 15:15:36.264918 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 12333
2026-09-26T15:15:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (629 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T15:15:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:15:36Z Disabled control of system clock
I20260926 15:15:36.341080 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44579
--webserver_interface=127.6.209.254
--webserver_port=0
--builtin_ntp_servers=127.6.209.212:32805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:44579
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435694426955-6983-0/testchainca.cert with env {}
W20260926 15:15:36.630438 12434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:36.630702 12434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:36.630847 12434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:36.637814 12434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:15:36.637924 12434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:36.637970 12434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:36.638041 12434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:15:36.638087 12434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 15:15:36.638147 12434 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 15:15:36.638195 12434 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 15:15:36.652665 12434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:32805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:44579
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435694426955-6983-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:36.654271 12434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:36.656100 12434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:36.666960 12442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:36.667088 12440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:36.667093 12439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:36.667795 12434 server_base.cc:1034] running on GCE node
I20260926 15:15:36.668470 12434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:36.669628 12434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:36.670795 12434 hybrid_clock.cc:648] HybridClock initialized: now 1790435736670758 us; error 52 us; skew 500 ppm
I20260926 15:15:36.671108 12434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:15:36.673085 12434 webserver.cc:460] Webserver started at http://127.6.209.254:45557/ using document root <none> and password file <none>
I20260926 15:15:36.673671 12434 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:36.673768 12434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:36.674044 12434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:36.676427 12434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "ea54549cb3524bf38bf13b2a9643ae50"
format_stamp: "Formatted at 2026-09-26 15:15:36 on dist-test-slave-j6q7"
server_key: "877cc77d379434fc0db33ab463db3f79"
server_key_iv: "195df013e245646817fc655f3080c31c"
server_key_version: "encryptionkey@0"
I20260926 15:15:36.677170 12434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "ea54549cb3524bf38bf13b2a9643ae50"
format_stamp: "Formatted at 2026-09-26 15:15:36 on dist-test-slave-j6q7"
server_key: "877cc77d379434fc0db33ab463db3f79"
server_key_iv: "195df013e245646817fc655f3080c31c"
server_key_version: "encryptionkey@0"
I20260926 15:15:36.681571 12434 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:15:36.684492 12448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:36.685658 12434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:15:36.685813 12434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "ea54549cb3524bf38bf13b2a9643ae50"
format_stamp: "Formatted at 2026-09-26 15:15:36 on dist-test-slave-j6q7"
server_key: "877cc77d379434fc0db33ab463db3f79"
server_key_iv: "195df013e245646817fc655f3080c31c"
server_key_version: "encryptionkey@0"
I20260926 15:15:36.686007 12434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:36.704420 12434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:36.705152 12434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:36.705402 12434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:36.721825 12434 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:44579
I20260926 15:15:36.721865 12499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:44579 every 8 connection(s)
I20260926 15:15:36.722945 12434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:15:36.725106 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 12434
I20260926 15:15:36.725296 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:15:36.725905 6983 external_mini_cluster.cc:1428] Setting key ad56ed571dbe1ed62799109e49f11553
I20260926 15:15:36.726235 12500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:36.737968 12503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:36.727524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45242 (local address 127.6.209.254:44579)
0926 15:15:36.728425 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:36.728434 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:36.728471 (+ 37us) server_negotiation.cc:407] Connection header received
0926 15:15:36.730107 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:36.730125 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:36.730515 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:36.730803 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:36.731338 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:36.732492 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:36.734225 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:36.734796 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:36.734919 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:36.735997 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:36.736003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:36.736013 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:15:36.736511 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:36.736579 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:15:36.736597 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:36.736676 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:36.736931 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 15:15:36.737213 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":341,"threads_started":1}
I20260926 15:15:36.738126 12500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea54549cb3524bf38bf13b2a9643ae50: Bootstrap starting.
I20260926 15:15:36.741303 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:15:36.741355 12500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea54549cb3524bf38bf13b2a9643ae50: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:36.742257 12500 log.cc:824] T 00000000000000000000000000000000 P ea54549cb3524bf38bf13b2a9643ae50: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:36.744753 12500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea54549cb3524bf38bf13b2a9643ae50: No bootstrap required, opened a new log
I20260926 15:15:36.751233 12500 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea54549cb3524bf38bf13b2a9643ae50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea54549cb3524bf38bf13b2a9643ae50" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44579 } }
I20260926 15:15:36.751513 12500 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea54549cb3524bf38bf13b2a9643ae50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:36.751622 12500 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea54549cb3524bf38bf13b2a9643ae50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea54549cb3524bf38bf13b2a9643ae50, State: Initialized, Role: FOLLOWER
I20260926 15:15:36.752146 12500 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea54549cb3524bf38bf13b2a9643ae50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea54549cb3524bf38bf13b2a9643ae50" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44579 } }
I20260926 15:15:36.752336 12500 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea54549cb3524bf38bf13b2a9643ae50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:15:36.752477 12500 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea54549cb3524bf38bf13b2a9643ae50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:15:36.752617 12500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea54549cb3524bf38bf13b2a9643ae50 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:36.754369 12500 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea54549cb3524bf38bf13b2a9643ae50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea54549cb3524bf38bf13b2a9643ae50" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44579 } }
I20260926 15:15:36.754792 12500 leader_election.cc:302] T 00000000000000000000000000000000 P ea54549cb3524bf38bf13b2a9643ae50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea54549cb3524bf38bf13b2a9643ae50; no voters:
I20260926 15:15:36.755286 12500 leader_election.cc:288] T 00000000000000000000000000000000 P ea54549cb3524bf38bf13b2a9643ae50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:15:36.755429 12505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea54549cb3524bf38bf13b2a9643ae50 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:36.756106 12505 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea54549cb3524bf38bf13b2a9643ae50 [term 1 LEADER]: Becoming Leader. State: Replica: ea54549cb3524bf38bf13b2a9643ae50, State: Running, Role: LEADER
I20260926 15:15:36.756646 12505 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea54549cb3524bf38bf13b2a9643ae50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea54549cb3524bf38bf13b2a9643ae50" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44579 } }
I20260926 15:15:36.757236 12500 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea54549cb3524bf38bf13b2a9643ae50 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:15:36.759505 12506 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea54549cb3524bf38bf13b2a9643ae50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea54549cb3524bf38bf13b2a9643ae50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea54549cb3524bf38bf13b2a9643ae50" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44579 } } }
I20260926 15:15:36.759476 12507 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea54549cb3524bf38bf13b2a9643ae50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea54549cb3524bf38bf13b2a9643ae50. Latest consensus state: current_term: 1 leader_uuid: "ea54549cb3524bf38bf13b2a9643ae50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea54549cb3524bf38bf13b2a9643ae50" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44579 } } }
I20260926 15:15:36.759879 12506 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea54549cb3524bf38bf13b2a9643ae50 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:36.759948 12507 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea54549cb3524bf38bf13b2a9643ae50 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:36.761502 12510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:15:36.767130 12510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:15:36.775223 12510 catalog_manager.cc:1347] Generated new cluster ID: a984e79559d64c119a30f9d8a0fd4d85
I20260926 15:15:36.775329 12510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:15:36.785470 12510 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:15:36.786545 12510 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:15:36.796440 12510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea54549cb3524bf38bf13b2a9643ae50: Generated new TSK 0
I20260926 15:15:36.797194 12510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:15:37.054585 12503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:36.810677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45254 (local address 127.6.209.254:44579)
0926 15:15:36.810823 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:36.810830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:36.811124 (+ 294us) server_negotiation.cc:407] Connection header received
0926 15:15:36.811330 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:36.811334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:36.811393 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:36.811517 (+ 124us) server_negotiation.cc:226] Negotiated authn=JWT
0926 15:15:36.811920 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:36.812516 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:36.813708 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:36.814105 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:36.814229 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:36.814633 (+ 404us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 15:15:37.053942 (+239309us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 15:15:37.054307 (+ 365us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45254: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 15:15:37.054711 12503 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45254: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 15:15:37.054883 12529 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:15:36.810592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44579 (local address 127.0.0.1:45254)
0926 15:15:36.811045 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:36.811057 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:36.811186 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:36.811523 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:36.811528 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:36.811538 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0926 15:15:36.811798 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:36.811805 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:36.812632 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:36.812639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:36.813592 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:36.813601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:36.814243 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:36.814249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:36.814389 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:36.814427 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 15:15:37.054218 (+239791us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 15:15:37.054464 (+ 246us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.209.254:44579: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":253,"threads_started":1}
I20260926 15:15:37.055889 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 12434
I20260926 15:15:37.057500 12495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:37.152848 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 12434
2026-09-26T15:15:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (869 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (364 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (369 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:15:37 dist-test-slave-j6q7 krb5kdc[12544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:15:37 dist-test-slave-j6q7 krb5kdc[12544](info): set up 2 sockets
Sep 26 15:15:37 dist-test-slave-j6q7 krb5kdc[12544](info): commencing operation
krb5kdc: starting...
W20260926 15:15:38.902271 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:15:39 dist-test-slave-j6q7 krb5kdc[12544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:15:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:15:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:15:39.065104 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:36749
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:36749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:39.341250 12557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:39.341535 12557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:39.341684 12557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:39.347864 12557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:15:39.347971 12557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:39.348065 12557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:39.348152 12557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:15:39.348217 12557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:15:39.356256 12557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:36749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:36749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:39.357690 12557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:39.358824 12557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:39.369910 12562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:39.369897 12565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:39.369933 12563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:39.370669 12557 server_base.cc:1034] running on GCE node
I20260926 15:15:39.371280 12557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:39.372364 12557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:39.373565 12557 hybrid_clock.cc:648] HybridClock initialized: now 1790435739373510 us; error 68 us; skew 500 ppm
I20260926 15:15:39.373867 12557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:39 dist-test-slave-j6q7 krb5kdc[12544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435739, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:39.376570 12557 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:15:39.377889 12557 webserver.cc:460] Webserver started at http://127.6.209.254:37937/ using document root <none> and password file <none>
I20260926 15:15:39.378427 12557 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:39.378542 12557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:39.378793 12557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:39.381363 12557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "b430654f7a344161a0d82423a006b392"
format_stamp: "Formatted at 2026-09-26 15:15:39 on dist-test-slave-j6q7"
server_key: "69dcefc65f9a531789ee36b062dd44ce"
server_key_iv: "885f65e56f684b16c778d5f1f1fe7b46"
server_key_version: "encryptionkey@0"
I20260926 15:15:39.382104 12557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "b430654f7a344161a0d82423a006b392"
format_stamp: "Formatted at 2026-09-26 15:15:39 on dist-test-slave-j6q7"
server_key: "69dcefc65f9a531789ee36b062dd44ce"
server_key_iv: "885f65e56f684b16c778d5f1f1fe7b46"
server_key_version: "encryptionkey@0"
I20260926 15:15:39.386471 12557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:15:39.389549 12572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:39.390537 12557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:15:39.390683 12557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "b430654f7a344161a0d82423a006b392"
format_stamp: "Formatted at 2026-09-26 15:15:39 on dist-test-slave-j6q7"
server_key: "69dcefc65f9a531789ee36b062dd44ce"
server_key_iv: "885f65e56f684b16c778d5f1f1fe7b46"
server_key_version: "encryptionkey@0"
I20260926 15:15:39.390841 12557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:39.409478 12557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:39.412483 12557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:39.412701 12557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:39.427103 12557 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:36749
I20260926 15:15:39.427170 12623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:36749 every 8 connection(s)
I20260926 15:15:39.428202 12557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:15:39.431023 12624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:39.438177 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 12557
I20260926 15:15:39.438403 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:15:39.438956 6983 external_mini_cluster.cc:1428] Setting key 43f6c5ec75b0793da3c41c9a48f76ee4
I20260926 15:15:39.439383 12624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b430654f7a344161a0d82423a006b392: Bootstrap starting.
I20260926 15:15:39.442579 12624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b430654f7a344161a0d82423a006b392: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:39.443496 12624 log.cc:824] T 00000000000000000000000000000000 P b430654f7a344161a0d82423a006b392: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:39.445953 12624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b430654f7a344161a0d82423a006b392: No bootstrap required, opened a new log
Sep 26 15:15:39 dist-test-slave-j6q7 krb5kdc[12544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:39.453794 12624 raft_consensus.cc:348] T 00000000000000000000000000000000 P b430654f7a344161a0d82423a006b392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b430654f7a344161a0d82423a006b392" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 36749 } }
I20260926 15:15:39.454097 12624 raft_consensus.cc:374] T 00000000000000000000000000000000 P b430654f7a344161a0d82423a006b392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:39.454190 12624 raft_consensus.cc:729] T 00000000000000000000000000000000 P b430654f7a344161a0d82423a006b392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b430654f7a344161a0d82423a006b392, State: Initialized, Role: FOLLOWER
I20260926 15:15:39.454643 12624 consensus_queue.cc:260] T 00000000000000000000000000000000 P b430654f7a344161a0d82423a006b392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b430654f7a344161a0d82423a006b392" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 36749 } }
I20260926 15:15:39.454843 12624 raft_consensus.cc:388] T 00000000000000000000000000000000 P b430654f7a344161a0d82423a006b392 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:15:39.454977 12624 raft_consensus.cc:482] T 00000000000000000000000000000000 P b430654f7a344161a0d82423a006b392 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:15:39.455124 12624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b430654f7a344161a0d82423a006b392 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:39.457054 12624 raft_consensus.cc:504] T 00000000000000000000000000000000 P b430654f7a344161a0d82423a006b392 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b430654f7a344161a0d82423a006b392" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 36749 } }
I20260926 15:15:39.457563 12624 leader_election.cc:302] T 00000000000000000000000000000000 P b430654f7a344161a0d82423a006b392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b430654f7a344161a0d82423a006b392; no voters:
I20260926 15:15:39.458130 12624 leader_election.cc:288] T 00000000000000000000000000000000 P b430654f7a344161a0d82423a006b392 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:15:39.458192 12629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b430654f7a344161a0d82423a006b392 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:39.459162 12627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:39.440472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39118 (local address 127.6.209.254:36749)
0926 15:15:39.441206 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:39.441217 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:39.441251 (+ 34us) server_negotiation.cc:407] Connection header received
0926 15:15:39.442788 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:39.442807 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:39.443100 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:39.443359 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:39.444954 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:39.446122 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:39.447784 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:39.448371 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:39.448490 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:39.452397 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:39.452403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:39.452413 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:39.452442 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:39.455297 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:39.455868 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:39.455875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:39.455882 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:39.456038 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:39.456365 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:39.456372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:39.456375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:39.456854 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:39.457151 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:39.457490 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 15:15:39.457865 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":393,"threads_started":1}
I20260926 15:15:39.459918 12624 sys_catalog.cc:564] T 00000000000000000000000000000000 P b430654f7a344161a0d82423a006b392 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:15:39.460609 12629 raft_consensus.cc:686] T 00000000000000000000000000000000 P b430654f7a344161a0d82423a006b392 [term 1 LEADER]: Becoming Leader. State: Replica: b430654f7a344161a0d82423a006b392, State: Running, Role: LEADER
I20260926 15:15:39.461195 12629 consensus_queue.cc:237] T 00000000000000000000000000000000 P b430654f7a344161a0d82423a006b392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b430654f7a344161a0d82423a006b392" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 36749 } }
I20260926 15:15:39.464263 12631 sys_catalog.cc:455] T 00000000000000000000000000000000 P b430654f7a344161a0d82423a006b392 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b430654f7a344161a0d82423a006b392. Latest consensus state: current_term: 1 leader_uuid: "b430654f7a344161a0d82423a006b392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b430654f7a344161a0d82423a006b392" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 36749 } } }
I20260926 15:15:39.464246 12630 sys_catalog.cc:455] T 00000000000000000000000000000000 P b430654f7a344161a0d82423a006b392 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b430654f7a344161a0d82423a006b392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b430654f7a344161a0d82423a006b392" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 36749 } } }
I20260926 15:15:39.464648 12631 sys_catalog.cc:458] T 00000000000000000000000000000000 P b430654f7a344161a0d82423a006b392 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:39.464660 12630 sys_catalog.cc:458] T 00000000000000000000000000000000 P b430654f7a344161a0d82423a006b392 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:39.466195 12639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:15:39.471902 12639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:15:39.479362 12639 catalog_manager.cc:1347] Generated new cluster ID: 2323a2aa150541a48bd37f779d0fe7f5
I20260926 15:15:39.479442 12639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:15:39.499514 12639 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:15:39.501122 12639 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:15:39.508606 12639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b430654f7a344161a0d82423a006b392: Generated new TSK 0
I20260926 15:15:39.509361 12639 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
I20260926 15:15:39.570077 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:36749
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:39.939384 12650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:39.939738 12650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:39.939911 12650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:39.948246 12650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:39.948340 12650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:39.948508 12650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:15:39.957736 12650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:36749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:39.958901 12650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:39.960186 12650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:39.972941 12656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:39.972950 12658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:39.973366 12655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:39.973655 12650 server_base.cc:1034] running on GCE node
I20260926 15:15:39.974243 12650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:39.974993 12650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:39.976181 12650 hybrid_clock.cc:648] HybridClock initialized: now 1790435739976154 us; error 30 us; skew 500 ppm
I20260926 15:15:39.976445 12650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:39 dist-test-slave-j6q7 krb5kdc[12544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435739, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:39.979329 12650 init.cc:377] Logged in from keytab as kudu/127.6.209.193@KRBTEST.COM (short username kudu)
I20260926 15:15:39.980639 12650 webserver.cc:460] Webserver started at http://127.6.209.193:36467/ using document root <none> and password file <none>
I20260926 15:15:39.981308 12650 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:39.981432 12650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:39.981717 12650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:39.984339 12650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "d52154a199f64dafb13867f99f555a7e"
format_stamp: "Formatted at 2026-09-26 15:15:39 on dist-test-slave-j6q7"
server_key: "925b2b2a8e6645492d663a3cb3c45354"
server_key_iv: "0347580c7fbc8bb158982d3bcf27e4c4"
server_key_version: "encryptionkey@0"
I20260926 15:15:39.985097 12650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "d52154a199f64dafb13867f99f555a7e"
format_stamp: "Formatted at 2026-09-26 15:15:39 on dist-test-slave-j6q7"
server_key: "925b2b2a8e6645492d663a3cb3c45354"
server_key_iv: "0347580c7fbc8bb158982d3bcf27e4c4"
server_key_version: "encryptionkey@0"
I20260926 15:15:39.989810 12650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:15:39.992988 12665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:39.994323 12650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:15:39.994468 12650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "d52154a199f64dafb13867f99f555a7e"
format_stamp: "Formatted at 2026-09-26 15:15:39 on dist-test-slave-j6q7"
server_key: "925b2b2a8e6645492d663a3cb3c45354"
server_key_iv: "0347580c7fbc8bb158982d3bcf27e4c4"
server_key_version: "encryptionkey@0"
I20260926 15:15:39.994640 12650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:40.027493 12650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:40.031257 12650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:40.031548 12650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:40.032428 12650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:40.033720 12650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:40.033790 12650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:40.033954 12650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:40.034011 12650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:40.064392 12650 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:37423
I20260926 15:15:40.064481 12777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:37423 every 8 connection(s)
I20260926 15:15:40.065678 12650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:15:40.070304 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 12650
I20260926 15:15:40.070506 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:15:40.071082 6983 external_mini_cluster.cc:1428] Setting key b8710100a44c6f63074c101699ee797e
Sep 26 15:15:40 dist-test-slave-j6q7 krb5kdc[12544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435739, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:40.084867 12780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:40.067994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:42633 (local address 127.6.209.254:36749)
0926 15:15:40.068392 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:40.068396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:40.069220 (+ 824us) server_negotiation.cc:407] Connection header received
0926 15:15:40.071007 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:40.071012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:40.071095 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:40.071204 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:40.072739 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:40.073816 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:40.075291 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:40.075998 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:40.076137 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:40.079490 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:40.079495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:40.079499 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:40.079529 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:40.082018 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:40.082680 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:40.082687 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:40.082693 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:40.082818 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:40.083214 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:40.083220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:40.083223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:40.083484 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:40.083627 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:40.084272 (+ 645us) server_negotiation.cc:299] Negotiation successful
0926 15:15:40.084491 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":210,"threads_started":1}
I20260926 15:15:40.085891 12781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:40.068409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:36749 (local address 127.6.209.193:42633)
0926 15:15:40.069062 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:40.069098 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:40.070756 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:40.071429 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:40.071437 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:40.072027 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:40.072579 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:40.072603 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:40.073954 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:40.073957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:40.075153 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:40.075164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:40.076194 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:40.076199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:40.076382 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:40.077086 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:40.077112 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:40.079292 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:40.082152 (+ 2860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:40.082161 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:40.082177 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:40.082568 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:40.082922 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:40.082928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:40.082932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:40.083112 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:40.083605 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:40.083612 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:40.083938 (+ 326us) client_negotiation.cc:769] Sending connection context
0926 15:15:40.084192 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 15:15:40.084673 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":254,"threads_started":1}
I20260926 15:15:40.087421 12778 heartbeater.cc:347] Connected to a master server at 127.6.209.254:36749
I20260926 15:15:40.087708 12778 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:40.088390 12778 heartbeater.cc:511] Master 127.6.209.254:36749 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.209.194@KRBTEST.COM; defaulting to no policy
I20260926 15:15:40.090412 12589 ts_manager.cc:194] Registered new tserver with Master: d52154a199f64dafb13867f99f555a7e (127.6.209.193:37423)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.194@KRBTEST.COM" created.
I20260926 15:15:40.091893 12589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.193@KRBTEST.COM'} at 127.6.209.193:42633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
I20260926 15:15:40.113785 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:36749
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:40.401141 12784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:40.401388 12784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:40.401558 12784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:40.407931 12784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:40.408092 12784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:40.408281 12784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:15:40.416740 12784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:36749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:40.417860 12784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:40.418957 12784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:40.430611 12789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:40.430642 12790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:40.431120 12792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:40.431885 12784 server_base.cc:1034] running on GCE node
I20260926 15:15:40.432336 12784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:40.433169 12784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:40.434373 12784 hybrid_clock.cc:648] HybridClock initialized: now 1790435740434329 us; error 44 us; skew 500 ppm
I20260926 15:15:40.434626 12784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:40 dist-test-slave-j6q7 krb5kdc[12544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435740, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:40.437404 12784 init.cc:377] Logged in from keytab as kudu/127.6.209.194@KRBTEST.COM (short username kudu)
I20260926 15:15:40.438755 12784 webserver.cc:460] Webserver started at http://127.6.209.194:38175/ using document root <none> and password file <none>
I20260926 15:15:40.439289 12784 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:40.439417 12784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:40.439678 12784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:40.442185 12784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "328433efc8f040aeb1a782eb47794cee"
format_stamp: "Formatted at 2026-09-26 15:15:40 on dist-test-slave-j6q7"
server_key: "94e228d7c9339415630780733a08eb5a"
server_key_iv: "69cb5dab343145a153cea750c44865f9"
server_key_version: "encryptionkey@0"
I20260926 15:15:40.442876 12784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "328433efc8f040aeb1a782eb47794cee"
format_stamp: "Formatted at 2026-09-26 15:15:40 on dist-test-slave-j6q7"
server_key: "94e228d7c9339415630780733a08eb5a"
server_key_iv: "69cb5dab343145a153cea750c44865f9"
server_key_version: "encryptionkey@0"
I20260926 15:15:40.447077 12784 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 15:15:40.450028 12799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:40.451207 12784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 15:15:40.451356 12784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "328433efc8f040aeb1a782eb47794cee"
format_stamp: "Formatted at 2026-09-26 15:15:40 on dist-test-slave-j6q7"
server_key: "94e228d7c9339415630780733a08eb5a"
server_key_iv: "69cb5dab343145a153cea750c44865f9"
server_key_version: "encryptionkey@0"
I20260926 15:15:40.451543 12784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:40.462008 12784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:40.465845 12784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:40.466100 12784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:40.466909 12784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:40.468112 12784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:40.468205 12784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:40.468335 12784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:40.468400 12784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:40.498867 12784 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:45169
I20260926 15:15:40.498966 12911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:45169 every 8 connection(s)
I20260926 15:15:40.499982 12784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:15:40.509245 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 12784
I20260926 15:15:40.509442 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:15:40.509837 6983 external_mini_cluster.cc:1428] Setting key bec802fde319be3f492daa591022c170
Sep 26 15:15:40 dist-test-slave-j6q7 krb5kdc[12544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435740, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:40.517633 12780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:40.502255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:36397 (local address 127.6.209.254:36749)
0926 15:15:40.502409 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:40.502413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:40.503375 (+ 962us) server_negotiation.cc:407] Connection header received
0926 15:15:40.504960 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:40.504965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:40.505044 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:40.505138 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:40.506395 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:40.507110 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:40.508526 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:40.509060 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:40.509201 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:40.512582 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:40.512586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:40.512589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:40.512615 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:40.514933 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:40.515532 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:40.515539 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:40.515542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:40.515653 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:40.516020 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:40.516027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:40.516030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:40.516248 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:40.516381 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:40.517094 (+ 713us) server_negotiation.cc:299] Negotiation successful
0926 15:15:40.517314 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 15:15:40.518488 12914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:40.502636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:36749 (local address 127.6.209.194:36397)
0926 15:15:40.503246 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:40.503279 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:40.504688 (+ 1409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:40.505339 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:40.505346 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:40.505766 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:40.506252 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:40.506267 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:40.507245 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:40.507250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:40.508403 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:40.508413 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:40.509263 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:40.509271 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:40.509493 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:40.510152 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:40.510180 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:40.512399 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:40.515053 (+ 2654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:40.515061 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:40.515074 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:40.515430 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:40.515747 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:40.515752 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:40.515755 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:40.515928 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:40.516365 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:40.516372 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:40.516710 (+ 338us) client_negotiation.cc:769] Sending connection context
0926 15:15:40.517020 (+ 310us) client_negotiation.cc:240] Negotiation successful
0926 15:15:40.517478 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":252,"threads_started":1}
I20260926 15:15:40.519699 12912 heartbeater.cc:347] Connected to a master server at 127.6.209.254:36749
I20260926 15:15:40.519979 12912 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:40.520689 12912 heartbeater.cc:511] Master 127.6.209.254:36749 requested a full tablet report, sending...
I20260926 15:15:40.522187 12589 ts_manager.cc:194] Registered new tserver with Master: 328433efc8f040aeb1a782eb47794cee (127.6.209.194:45169)
I20260926 15:15:40.523373 12589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.194@KRBTEST.COM'} at 127.6.209.194:36397
WARNING: no policy specified for kudu/127.6.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
I20260926 15:15:40.549618 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:36749
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:40.849864 12917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:40.850088 12917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:40.850176 12917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:40.856745 12917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:40.856848 12917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:40.856961 12917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:15:40.865918 12917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:36749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:40.866937 12917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:40.868129 12917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:40.880226 12922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:40.880260 12925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:40.880280 12923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:40.881006 12917 server_base.cc:1034] running on GCE node
I20260926 15:15:40.881528 12917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:40.882391 12917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:40.883611 12917 hybrid_clock.cc:648] HybridClock initialized: now 1790435740883569 us; error 55 us; skew 500 ppm
I20260926 15:15:40.883888 12917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:40 dist-test-slave-j6q7 krb5kdc[12544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435740, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:40.886790 12917 init.cc:377] Logged in from keytab as kudu/127.6.209.195@KRBTEST.COM (short username kudu)
I20260926 15:15:40.888037 12917 webserver.cc:460] Webserver started at http://127.6.209.195:39743/ using document root <none> and password file <none>
I20260926 15:15:40.888609 12917 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:40.888731 12917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:40.889117 12917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:40.891732 12917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "91eac02ce8cc43d2adfb0c6141789809"
format_stamp: "Formatted at 2026-09-26 15:15:40 on dist-test-slave-j6q7"
server_key: "b6ff818f4e55bed81d84e6378f5f4a46"
server_key_iv: "8580a427e40d2dc41f6c09baa09257ff"
server_key_version: "encryptionkey@0"
I20260926 15:15:40.892480 12917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "91eac02ce8cc43d2adfb0c6141789809"
format_stamp: "Formatted at 2026-09-26 15:15:40 on dist-test-slave-j6q7"
server_key: "b6ff818f4e55bed81d84e6378f5f4a46"
server_key_iv: "8580a427e40d2dc41f6c09baa09257ff"
server_key_version: "encryptionkey@0"
I20260926 15:15:40.897209 12917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260926 15:15:40.900327 12932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:40.901473 12917 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:15:40.901630 12917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "91eac02ce8cc43d2adfb0c6141789809"
format_stamp: "Formatted at 2026-09-26 15:15:40 on dist-test-slave-j6q7"
server_key: "b6ff818f4e55bed81d84e6378f5f4a46"
server_key_iv: "8580a427e40d2dc41f6c09baa09257ff"
server_key_version: "encryptionkey@0"
I20260926 15:15:40.901827 12917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:40.926749 12917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:40.930681 12917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:40.930934 12917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:40.931823 12917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:40.933069 12917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:40.933161 12917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:40.933284 12917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:40.933373 12917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:40.964432 12917 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:37197
I20260926 15:15:40.964481 13044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:37197 every 8 connection(s)
I20260926 15:15:40.965667 12917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:15:40.974474 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 12917
I20260926 15:15:40.974647 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:15:40.975041 6983 external_mini_cluster.cc:1428] Setting key 9cd5aba5647f94f237aecc1da575606c
Sep 26 15:15:40 dist-test-slave-j6q7 krb5kdc[12544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435740, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:40.983858 12780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:40.968020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:35777 (local address 127.6.209.254:36749)
0926 15:15:40.968174 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:40.968179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:40.969329 (+ 1150us) server_negotiation.cc:407] Connection header received
0926 15:15:40.971064 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:40.971069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:40.971149 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:40.971260 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:40.972634 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:40.973427 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:40.974943 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:40.975379 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:40.975485 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:40.978945 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:40.978951 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:40.978955 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:40.978992 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:40.981199 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:40.981857 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:40.981865 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:40.981871 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:40.982002 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:40.982396 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:40.982400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:40.982403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:40.982630 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:40.982756 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:40.983351 (+ 595us) server_negotiation.cc:299] Negotiation successful
0926 15:15:40.983539 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 15:15:40.984701 13047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:40.968515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:36749 (local address 127.6.209.195:35777)
0926 15:15:40.969185 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:40.969225 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:40.970779 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:40.971476 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:40.971483 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:40.971954 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:40.972478 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:40.972494 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:40.973572 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:40.973577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:40.974808 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:40.974819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:40.975576 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:40.975581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:40.975811 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:40.976451 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:40.976470 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:40.978771 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:40.981321 (+ 2550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:40.981331 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:40.981347 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:40.981727 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:40.982114 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:40.982120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:40.982135 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:40.982285 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:40.982740 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:40.982746 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:40.983009 (+ 263us) client_negotiation.cc:769] Sending connection context
0926 15:15:40.983260 (+ 251us) client_negotiation.cc:240] Negotiation successful
0926 15:15:40.983706 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":295,"threads_started":1}
I20260926 15:15:40.985826 13045 heartbeater.cc:347] Connected to a master server at 127.6.209.254:36749
I20260926 15:15:40.986114 13045 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:40.986783 13045 heartbeater.cc:511] Master 127.6.209.254:36749 requested a full tablet report, sending...
I20260926 15:15:40.988006 12589 ts_manager.cc:194] Registered new tserver with Master: 91eac02ce8cc43d2adfb0c6141789809 (127.6.209.195:37197)
I20260926 15:15:40.988694 12589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.195@KRBTEST.COM'} at 127.6.209.195:35777
I20260926 15:15:40.989470 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/test-admin.keytab.
Sep 26 15:15:41 dist-test-slave-j6q7 krb5kdc[12544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:41.013597 6983 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 15:15:41.017936 13052 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:15:41.015900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.6.209.193:37423)
0926 15:15:41.016393 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:41.016400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:41.016425 (+ 25us) server_negotiation.cc:407] Connection header received
0926 15:15:41.016479 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:41.016485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:41.016694 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:41.016885 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:41.017532 (+ 647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59658: BlockingRecv error: recv got EOF from 127.0.0.1:59658 (error 108)
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":270,"threads_started":1}
W20260926 15:15:41.017984 6983 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/krb5cc' of type FILE
Sep 26 15:15:41 dist-test-slave-j6q7 krb5kdc[12544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:41.019258 6983 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 15:15:41 dist-test-slave-j6q7 krb5kdc[12544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:41.020351 6983 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 15:15:41 dist-test-slave-j6q7 krb5kdc[12544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.193@KRBTEST.COM
I20260926 15:15:41.033439 13052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:41.021817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59668 (local address 127.6.209.193:37423)
0926 15:15:41.021975 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:41.021981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:41.022074 (+ 93us) server_negotiation.cc:407] Connection header received
0926 15:15:41.022258 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:41.022263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:41.022315 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:41.022396 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:41.023212 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:41.024076 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:41.025160 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:41.025641 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:41.025762 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:41.028972 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:41.028980 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:41.028992 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:41.029021 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:41.031369 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:41.031847 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:41.031852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:41.031855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:41.031957 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:41.032219 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:41.032222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:41.032224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:41.032510 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:41.032712 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:41.033015 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 15:15:41.033192 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 15:15:41.037978 13052 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:15:41.036580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.6.209.193:37423)
0926 15:15:41.036774 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:41.036779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:41.037031 (+ 252us) server_negotiation.cc:407] Connection header received
0926 15:15:41.037132 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:41.037135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:41.037180 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:41.037243 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:41.037811 (+ 568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59672: BlockingRecv error: recv got EOF from 127.0.0.1:59672 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 15:15:41.038161 6983 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/krb5cc' of type FILE
Sep 26 15:15:41 dist-test-slave-j6q7 krb5kdc[12544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:41.039315 6983 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 15:15:41 dist-test-slave-j6q7 krb5kdc[12544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:41.040340 6983 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 15:15:41 dist-test-slave-j6q7 krb5kdc[12544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.193@KRBTEST.COM
I20260926 15:15:41.051211 13052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:41.041392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59684 (local address 127.6.209.193:37423)
0926 15:15:41.041565 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:41.041572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:41.041791 (+ 219us) server_negotiation.cc:407] Connection header received
0926 15:15:41.041955 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:41.041964 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:41.042010 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:41.042095 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:41.042867 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:41.043438 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:41.044401 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:41.044733 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:41.044893 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:41.047656 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:41.047661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:41.047664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:41.047696 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:41.049706 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:41.050135 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:41.050138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:41.050140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:41.050222 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:41.050460 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:41.050463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:41.050464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:41.050642 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:41.050793 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:41.050942 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 15:15:41.051053 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 15:15:41.054693 13052 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:15:41.053326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59700 (local address 127.6.209.193:37423)
0926 15:15:41.053472 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:41.053478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:41.053596 (+ 118us) server_negotiation.cc:407] Connection header received
0926 15:15:41.053736 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:41.053740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:41.053818 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:41.053904 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:41.054492 (+ 588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59700: BlockingRecv error: recv got EOF from 127.0.0.1:59700 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 15:15:41.055085 6983 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435694426955-6983-0/krb5kdc/krb5cc)
Sep 26 15:15:41 dist-test-slave-j6q7 krb5kdc[12544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:41.056319 6983 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 15:15:41 dist-test-slave-j6q7 krb5kdc[12544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:41.057505 6983 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 15:15:41 dist-test-slave-j6q7 krb5kdc[12544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.193@KRBTEST.COM
I20260926 15:15:41.069195 13052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:41.058431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59708 (local address 127.6.209.193:37423)
0926 15:15:41.058583 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:41.058588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:41.058770 (+ 182us) server_negotiation.cc:407] Connection header received
0926 15:15:41.058860 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:41.058863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:41.058901 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:41.058961 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:41.059682 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:41.060186 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:41.061253 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:41.061626 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:41.061711 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:41.064540 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:41.064543 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:41.064546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:41.064567 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:41.067026 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:41.067671 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:41.067674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:41.067676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:41.067798 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:41.068086 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:41.068089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:41.068090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:41.068312 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:41.068474 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:41.068731 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 15:15:41.068909 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 15:15:41.070399 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 12650
I20260926 15:15:41.079790 13052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:41.071014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.6.209.193:37423)
0926 15:15:41.071158 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:41.071165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:41.071437 (+ 272us) server_negotiation.cc:407] Connection header received
0926 15:15:41.071548 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:41.071551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:41.071590 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:41.071649 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:41.072461 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:41.073014 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:41.073921 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:41.074457 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:41.074556 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:41.075979 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:41.075983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:41.075985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:41.076006 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:41.078172 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:41.078597 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:41.078600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:41.078601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:41.078699 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:41.078936 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:41.078938 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:41.078940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:41.079107 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:41.079246 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:41.079423 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 15:15:41.079531 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 15:15:41.080178 12773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:41.152802 12778 heartbeater.cc:503] Master 127.6.209.254:36749 was elected leader, sending a full tablet report...
I20260926 15:15:41.153162 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 12650
W20260926 15:15:41.168124 12576 connection.cc:535] server connection from 127.6.209.193:42633 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 15:15:41.168447 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 12784
Sep 26 15:15:41 dist-test-slave-j6q7 krb5kdc[12544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.194@KRBTEST.COM
I20260926 15:15:41.182180 13065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:41.169323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35554 (local address 127.6.209.194:45169)
0926 15:15:41.169826 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:41.169832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:41.169853 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:15:41.169902 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:41.169908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:41.170098 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:41.170248 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:41.171092 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:41.172003 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:41.173189 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:41.173708 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:41.173837 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:41.177092 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:41.177104 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:41.177123 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:41.177150 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:41.179809 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:41.180353 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:41.180360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:41.180363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:41.180486 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:41.180827 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:41.180832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:41.180838 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:41.181151 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:41.181436 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:41.181692 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 15:15:41.181904 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":268,"threads_started":1}
I20260926 15:15:41.183358 12907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:41.254667 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 12784
I20260926 15:15:41.268510 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 12917
Sep 26 15:15:41 dist-test-slave-j6q7 krb5kdc[12544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.195@KRBTEST.COM
I20260926 15:15:41.281416 13067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:41.269290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56016 (local address 127.6.209.195:37197)
0926 15:15:41.269765 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:41.269772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:41.269804 (+ 32us) server_negotiation.cc:407] Connection header received
0926 15:15:41.269855 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:41.269860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:41.270047 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:41.270155 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:41.270927 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:41.271804 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:41.272802 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:41.273387 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:41.273472 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:41.276709 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:41.276719 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:41.276731 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:41.276762 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:41.279375 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:41.279831 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:41.279836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:41.279838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:41.279929 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:41.280192 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:41.280194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:41.280196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:41.280460 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:41.280643 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:41.280865 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 15:15:41.281059 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":220,"threads_started":1}
I20260926 15:15:41.282509 13040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:41.355142 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 12917
I20260926 15:15:41.369627 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 12557
I20260926 15:15:41.370563 12619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:41.460630 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 12557
2026-09-26T15:15:41Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3576 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:15:41 dist-test-slave-j6q7 krb5kdc[13072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:15:41 dist-test-slave-j6q7 krb5kdc[13072](info): set up 2 sockets
Sep 26 15:15:41 dist-test-slave-j6q7 krb5kdc[13072](info): commencing operation
krb5kdc: starting...
W20260926 15:15:42.445044 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:15:42 dist-test-slave-j6q7 krb5kdc[13072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:15:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:15:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.254.keytab.
Entry for principal oryx/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.254.keytab.
I20260926 15:15:42.609702 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44331
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.254.keytab
--principal=oryx/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:38737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:44331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:42.902271 13085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:42.902496 13085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:42.902639 13085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:42.908946 13085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:15:42.909027 13085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:42.909070 13085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:42.909122 13085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:15:42.909180 13085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:15:42.918818 13085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:38737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:44331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.254.keytab
--principal=oryx/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:42.919831 13085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:42.921031 13085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:42.931326 13091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:42.931370 13090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:42.931391 13093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:42.932224 13085 server_base.cc:1034] running on GCE node
I20260926 15:15:42.932971 13085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:42.934155 13085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:42.935356 13085 hybrid_clock.cc:648] HybridClock initialized: now 1790435742935325 us; error 39 us; skew 500 ppm
I20260926 15:15:42.935638 13085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:42 dist-test-slave-j6q7 krb5kdc[13072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435742, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:42.938364 13085 init.cc:377] Logged in from keytab as oryx/127.6.209.254@KRBTEST.COM (short username oryx)
I20260926 15:15:42.939574 13085 webserver.cc:460] Webserver started at http://127.6.209.254:36201/ using document root <none> and password file <none>
I20260926 15:15:42.940142 13085 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:42.940272 13085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:42.940518 13085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:42.943161 13085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "487b6fe6e7264e4db494c8057789998b"
format_stamp: "Formatted at 2026-09-26 15:15:42 on dist-test-slave-j6q7"
server_key: "d4b8a6d8501657c9b7ba9bf5d15ef5c4"
server_key_iv: "848729a1c20c4084fc3936b689c7b07a"
server_key_version: "encryptionkey@0"
I20260926 15:15:42.943810 13085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "487b6fe6e7264e4db494c8057789998b"
format_stamp: "Formatted at 2026-09-26 15:15:42 on dist-test-slave-j6q7"
server_key: "d4b8a6d8501657c9b7ba9bf5d15ef5c4"
server_key_iv: "848729a1c20c4084fc3936b689c7b07a"
server_key_version: "encryptionkey@0"
I20260926 15:15:42.948246 13085 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 15:15:42.950929 13100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:42.952035 13085 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 15:15:42.952183 13085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "487b6fe6e7264e4db494c8057789998b"
format_stamp: "Formatted at 2026-09-26 15:15:42 on dist-test-slave-j6q7"
server_key: "d4b8a6d8501657c9b7ba9bf5d15ef5c4"
server_key_iv: "848729a1c20c4084fc3936b689c7b07a"
server_key_version: "encryptionkey@0"
I20260926 15:15:42.952354 13085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:42.972096 13085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:42.975309 13085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:42.975536 13085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:42.991333 13085 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:44331
I20260926 15:15:42.991364 13151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:44331 every 8 connection(s)
I20260926 15:15:42.992537 13085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:15:42.994546 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 13085
I20260926 15:15:42.994764 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:15:42.995573 6983 external_mini_cluster.cc:1428] Setting key fe928cf27a3c7de39d90b1dffb74dfee
I20260926 15:15:42.996402 13152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:43.006922 13152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b: Bootstrap starting.
Sep 26 15:15:43 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM
I20260926 15:15:43.009887 13152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:43.010763 13152 log.cc:824] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:43.013170 13152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b: No bootstrap required, opened a new log
I20260926 15:15:43.015821 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:42.997738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36104 (local address 127.6.209.254:44331)
0926 15:15:42.998595 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:42.998604 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:42.998639 (+ 35us) server_negotiation.cc:407] Connection header received
0926 15:15:43.000694 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:43.000715 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:43.001158 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:43.001508 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:43.002790 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:43.003919 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:43.004960 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:43.005519 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:43.005669 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:43.009507 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:43.009514 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:43.009526 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:43.009565 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:43.012470 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:43.013106 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:43.013111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:43.013115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:43.013267 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:43.013671 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:43.013674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:43.013676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:43.014081 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:43.014267 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:43.014640 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 15:15:43.014964 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":437,"threads_started":1}
I20260926 15:15:43.020349 13152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } }
I20260926 15:15:43.020627 13152 raft_consensus.cc:374] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:43.020716 13152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 487b6fe6e7264e4db494c8057789998b, State: Initialized, Role: FOLLOWER
I20260926 15:15:43.021204 13152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } }
I20260926 15:15:43.021418 13152 raft_consensus.cc:388] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:15:43.021538 13152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:15:43.021673 13152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:43.023313 13152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } }
I20260926 15:15:43.023716 13152 leader_election.cc:302] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 487b6fe6e7264e4db494c8057789998b; no voters:
I20260926 15:15:43.024186 13152 leader_election.cc:288] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:15:43.024313 13157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:43.024911 13157 raft_consensus.cc:686] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [term 1 LEADER]: Becoming Leader. State: Replica: 487b6fe6e7264e4db494c8057789998b, State: Running, Role: LEADER
I20260926 15:15:43.025357 13157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } }
I20260926 15:15:43.025959 13152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:15:43.027587 13159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 487b6fe6e7264e4db494c8057789998b. Latest consensus state: current_term: 1 leader_uuid: "487b6fe6e7264e4db494c8057789998b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } }
I20260926 15:15:43.027911 13159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:43.027915 13158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "487b6fe6e7264e4db494c8057789998b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } }
I20260926 15:15:43.029492 13158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:43.029531 13162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:15:43.034183 13162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:15:43.041770 13162 catalog_manager.cc:1347] Generated new cluster ID: aaebb9b0e62d4ad5b164b4119414725a
I20260926 15:15:43.041878 13162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:15:43.052757 13162 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:15:43.054159 13162 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:15:43.060592 13162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b: Generated new TSK 0
I20260926 15:15:43.061434 13162 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.193.keytab.
Entry for principal oryx/127.6.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.193.keytab.
I20260926 15:15:43.112241 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44331
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.193.keytab
--principal=oryx/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:38737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:43.409389 13178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:43.409639 13178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:43.409848 13178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:43.416790 13178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:43.416934 13178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:43.417140 13178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:15:43.426561 13178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:38737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.193.keytab
--principal=oryx/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:43.427680 13178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:43.428963 13178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:43.441710 13186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:43.441659 13184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:43.442067 13183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:43.442368 13178 server_base.cc:1034] running on GCE node
I20260926 15:15:43.442967 13178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:43.443784 13178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:43.444954 13178 hybrid_clock.cc:648] HybridClock initialized: now 1790435743444917 us; error 30 us; skew 500 ppm
I20260926 15:15:43.445232 13178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:43 dist-test-slave-j6q7 krb5kdc[13072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435743, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:43.448105 13178 init.cc:377] Logged in from keytab as oryx/127.6.209.193@KRBTEST.COM (short username oryx)
I20260926 15:15:43.449352 13178 webserver.cc:460] Webserver started at http://127.6.209.193:32855/ using document root <none> and password file <none>
I20260926 15:15:43.449985 13178 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:43.450124 13178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:43.450398 13178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:43.452967 13178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "6dc3ad57bb66483bbc89eb29244df045"
format_stamp: "Formatted at 2026-09-26 15:15:43 on dist-test-slave-j6q7"
server_key: "0a732365984a31318527f72e36dfbdd5"
server_key_iv: "a0289cf6391ca25d9bb958ee687b179d"
server_key_version: "encryptionkey@0"
I20260926 15:15:43.453660 13178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "6dc3ad57bb66483bbc89eb29244df045"
format_stamp: "Formatted at 2026-09-26 15:15:43 on dist-test-slave-j6q7"
server_key: "0a732365984a31318527f72e36dfbdd5"
server_key_iv: "a0289cf6391ca25d9bb958ee687b179d"
server_key_version: "encryptionkey@0"
I20260926 15:15:43.458338 13178 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 15:15:43.461438 13193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:43.462570 13178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:15:43.462731 13178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "6dc3ad57bb66483bbc89eb29244df045"
format_stamp: "Formatted at 2026-09-26 15:15:43 on dist-test-slave-j6q7"
server_key: "0a732365984a31318527f72e36dfbdd5"
server_key_iv: "a0289cf6391ca25d9bb958ee687b179d"
server_key_version: "encryptionkey@0"
I20260926 15:15:43.462913 13178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:43.474998 13178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:43.478802 13178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:43.479063 13178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:43.479858 13178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:43.481035 13178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:43.481137 13178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:43.481277 13178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:43.481353 13178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:43.512169 13178 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:38531
I20260926 15:15:43.512187 13305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:38531 every 8 connection(s)
I20260926 15:15:43.513357 13178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:15:43.518169 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 13178
I20260926 15:15:43.518356 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:15:43.519052 6983 external_mini_cluster.cc:1428] Setting key 2059094fb2601b1baf0ddd041cf597ff
Sep 26 15:15:43 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435743, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM
I20260926 15:15:43.531636 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:43.515585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:50577 (local address 127.6.209.254:44331)
0926 15:15:43.515804 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:43.515810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:43.516849 (+ 1039us) server_negotiation.cc:407] Connection header received
0926 15:15:43.518590 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:43.518597 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:43.518673 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:43.518788 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:43.520154 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:43.521073 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:43.522457 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:43.523091 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:43.523214 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:43.526223 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:43.526227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:43.526229 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:43.526255 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:43.528766 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:43.529664 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:43.529667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:43.529669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:43.529789 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:43.530223 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:43.530230 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:43.530232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:43.530466 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:43.530559 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:43.531269 (+ 710us) server_negotiation.cc:299] Negotiation successful
0926 15:15:43.531411 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":82}
I20260926 15:15:43.532956 13308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:43.516031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.6.209.193:50577)
0926 15:15:43.516663 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:43.516703 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:43.518340 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:43.518939 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:43.518945 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:43.519435 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:43.520025 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:43.520040 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:43.521208 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:43.521212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:43.522321 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:43.522331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:43.523259 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:43.523263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:43.523415 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:43.524007 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:43.524026 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:43.526019 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:43.528970 (+ 2951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:43.528979 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:43.528994 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:43.529565 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:43.529888 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:43.529894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:43.529900 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:43.530111 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:43.530576 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:43.530585 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:43.530954 (+ 369us) client_negotiation.cc:769] Sending connection context
0926 15:15:43.531212 (+ 258us) client_negotiation.cc:240] Negotiation successful
0926 15:15:43.531679 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":240,"threads_started":1}
I20260926 15:15:43.534299 13306 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44331
I20260926 15:15:43.534560 13306 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:43.535238 13306 heartbeater.cc:511] Master 127.6.209.254:44331 requested a full tablet report, sending...
I20260926 15:15:43.537184 13117 ts_manager.cc:194] Registered new tserver with Master: 6dc3ad57bb66483bbc89eb29244df045 (127.6.209.193:38531)
WARNING: no policy specified for oryx/127.6.209.194@KRBTEST.COM; defaulting to no policy
I20260926 15:15:43.538559 13117 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.209.193@KRBTEST.COM'} at 127.6.209.193:50577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.194.keytab.
Entry for principal oryx/127.6.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.194.keytab.
I20260926 15:15:43.560106 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44331
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.194.keytab
--principal=oryx/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:38737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:43.853659 13311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:43.853947 13311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:43.854135 13311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:43.861003 13311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:43.861152 13311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:43.861325 13311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:15:43.869753 13311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:38737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.194.keytab
--principal=oryx/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:43.870800 13311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:43.872017 13311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:43.883965 13317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:43.884001 13319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:43.884187 13316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:43.884788 13311 server_base.cc:1034] running on GCE node
I20260926 15:15:43.885424 13311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:43.886233 13311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:43.887458 13311 hybrid_clock.cc:648] HybridClock initialized: now 1790435743887408 us; error 33 us; skew 500 ppm
I20260926 15:15:43.887768 13311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:43 dist-test-slave-j6q7 krb5kdc[13072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435743, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:43.890722 13311 init.cc:377] Logged in from keytab as oryx/127.6.209.194@KRBTEST.COM (short username oryx)
I20260926 15:15:43.891948 13311 webserver.cc:460] Webserver started at http://127.6.209.194:36145/ using document root <none> and password file <none>
I20260926 15:15:43.892554 13311 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:43.892691 13311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:43.893009 13311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:43.895588 13311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "2adbe25c005c4e93927b3d15df65f065"
format_stamp: "Formatted at 2026-09-26 15:15:43 on dist-test-slave-j6q7"
server_key: "f8a934ca2d19e76c159ef27ba1064ef8"
server_key_iv: "16b9e3f799f4bebc7c2c462b99dca788"
server_key_version: "encryptionkey@0"
I20260926 15:15:43.896364 13311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "2adbe25c005c4e93927b3d15df65f065"
format_stamp: "Formatted at 2026-09-26 15:15:43 on dist-test-slave-j6q7"
server_key: "f8a934ca2d19e76c159ef27ba1064ef8"
server_key_iv: "16b9e3f799f4bebc7c2c462b99dca788"
server_key_version: "encryptionkey@0"
I20260926 15:15:43.901260 13311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 15:15:43.904374 13326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:43.905652 13311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:15:43.905828 13311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "2adbe25c005c4e93927b3d15df65f065"
format_stamp: "Formatted at 2026-09-26 15:15:43 on dist-test-slave-j6q7"
server_key: "f8a934ca2d19e76c159ef27ba1064ef8"
server_key_iv: "16b9e3f799f4bebc7c2c462b99dca788"
server_key_version: "encryptionkey@0"
I20260926 15:15:43.906006 13311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:43.915654 13311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:43.920210 13311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:43.920433 13311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:43.921272 13311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:43.922416 13311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:43.922487 13311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:43.922570 13311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:43.922622 13311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:43.951687 13311 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:34425
I20260926 15:15:43.951727 13438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:34425 every 8 connection(s)
I20260926 15:15:43.952934 13311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:15:43.955189 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 13311
I20260926 15:15:43.955396 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:15:43.955832 6983 external_mini_cluster.cc:1428] Setting key d2831ee00733cd463fb4d8518b2c64d2
Sep 26 15:15:43 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435743, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM
I20260926 15:15:43.972381 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:43.955273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:42885 (local address 127.6.209.254:44331)
0926 15:15:43.955465 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:43.955471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:43.956665 (+ 1194us) server_negotiation.cc:407] Connection header received
0926 15:15:43.958484 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:43.958493 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:43.958581 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:43.958697 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:43.960023 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:43.961199 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:43.962641 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:43.963342 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:43.963490 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:43.967141 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:43.967146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:43.967149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:43.967179 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:43.969497 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:43.970164 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:43.970170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:43.970174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:43.970299 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:43.970706 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:43.970713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:43.970719 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:43.970965 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:43.971124 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:43.971766 (+ 642us) server_negotiation.cc:299] Negotiation successful
0926 15:15:43.971959 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 15:15:43.973268 13441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:43.955777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.6.209.194:42885)
0926 15:15:43.956531 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:43.956576 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:43.958228 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:43.958886 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:43.958893 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:43.959328 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:43.959837 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:43.959852 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:43.961354 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:43.961358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:43.962497 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:43.962507 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:43.963521 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:43.963525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:43.963745 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:43.964416 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:43.964442 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:43.966925 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:43.969633 (+ 2708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:43.969641 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:43.969654 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:43.970040 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:43.970408 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:43.970413 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:43.970417 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:43.970588 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:43.971088 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:43.971096 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:43.971408 (+ 312us) client_negotiation.cc:769] Sending connection context
0926 15:15:43.971665 (+ 257us) client_negotiation.cc:240] Negotiation successful
0926 15:15:43.972118 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":323,"threads_started":1}
I20260926 15:15:43.974552 13439 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44331
I20260926 15:15:43.974840 13439 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.6.209.195@KRBTEST.COM; defaulting to no policy
I20260926 15:15:43.975464 13439 heartbeater.cc:511] Master 127.6.209.254:44331 requested a full tablet report, sending...
I20260926 15:15:43.976971 13117 ts_manager.cc:194] Registered new tserver with Master: 2adbe25c005c4e93927b3d15df65f065 (127.6.209.194:34425)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.209.195@KRBTEST.COM" created.
I20260926 15:15:43.977842 13117 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.209.194@KRBTEST.COM'} at 127.6.209.194:42885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.195.keytab.
Entry for principal oryx/127.6.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.195.keytab.
I20260926 15:15:43.999418 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44331
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.195.keytab
--principal=oryx/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:38737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:44.285533 13444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:44.285777 13444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:44.285851 13444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:44.291826 13444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:44.291899 13444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:44.291983 13444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:15:44.300352 13444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:38737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.195.keytab
--principal=oryx/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:44.301380 13444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:44.302492 13444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:44.314447 13450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:44.314419 13452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:44.314693 13449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:44.315056 13444 server_base.cc:1034] running on GCE node
I20260926 15:15:44.315543 13444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:44.316275 13444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:44.317456 13444 hybrid_clock.cc:648] HybridClock initialized: now 1790435744317410 us; error 40 us; skew 500 ppm
I20260926 15:15:44.317691 13444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:44 dist-test-slave-j6q7 krb5kdc[13072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435744, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:44.320434 13444 init.cc:377] Logged in from keytab as oryx/127.6.209.195@KRBTEST.COM (short username oryx)
I20260926 15:15:44.321774 13444 webserver.cc:460] Webserver started at http://127.6.209.195:37689/ using document root <none> and password file <none>
I20260926 15:15:44.322381 13444 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:44.322496 13444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:44.322762 13444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:44.325387 13444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "ac66294fc8734840a60cbe11e4625260"
format_stamp: "Formatted at 2026-09-26 15:15:44 on dist-test-slave-j6q7"
server_key: "5cd3210ffe46dddb6ff3f19f7056510d"
server_key_iv: "ecf0ded2cd40b7b71c0598703c263374"
server_key_version: "encryptionkey@0"
I20260926 15:15:44.326030 13444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "ac66294fc8734840a60cbe11e4625260"
format_stamp: "Formatted at 2026-09-26 15:15:44 on dist-test-slave-j6q7"
server_key: "5cd3210ffe46dddb6ff3f19f7056510d"
server_key_iv: "ecf0ded2cd40b7b71c0598703c263374"
server_key_version: "encryptionkey@0"
I20260926 15:15:44.330415 13444 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 15:15:44.333223 13459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:44.334378 13444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260926 15:15:44.334520 13444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "ac66294fc8734840a60cbe11e4625260"
format_stamp: "Formatted at 2026-09-26 15:15:44 on dist-test-slave-j6q7"
server_key: "5cd3210ffe46dddb6ff3f19f7056510d"
server_key_iv: "ecf0ded2cd40b7b71c0598703c263374"
server_key_version: "encryptionkey@0"
I20260926 15:15:44.334684 13444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:44.374590 13444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:44.378494 13444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:44.378762 13444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:44.379592 13444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:44.380877 13444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:44.381012 13444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:44.381140 13444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:44.381212 13444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:44.414711 13444 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:44185
I20260926 15:15:44.414745 13571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:44185 every 8 connection(s)
I20260926 15:15:44.415763 13444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:15:44.425460 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 13444
I20260926 15:15:44.425647 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:15:44.426110 6983 external_mini_cluster.cc:1428] Setting key 76f90b25d46cf7f145d9dbb55a7c7b27
Sep 26 15:15:44 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435744, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM
I20260926 15:15:44.433069 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:44.418023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:35607 (local address 127.6.209.254:44331)
0926 15:15:44.418232 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:44.418238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:44.419218 (+ 980us) server_negotiation.cc:407] Connection header received
0926 15:15:44.420734 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:44.420738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:44.420884 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:44.420994 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:44.422282 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.422939 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.424307 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.424675 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.424758 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:44.428274 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:44.428279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:44.428284 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:44.428316 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:44.430732 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:44.431322 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:44.431326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:44.431328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:44.431456 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:44.431778 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:44.431781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:44.431783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:44.432012 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:44.432107 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:44.432596 (+ 489us) server_negotiation.cc:299] Negotiation successful
0926 15:15:44.432793 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 15:15:44.433877 13574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:44.418517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.6.209.195:35607)
0926 15:15:44.419086 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:44.419117 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:44.420525 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:44.421178 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:44.421185 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:44.421638 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:44.422133 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:44.422161 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.423063 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.423066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:44.424191 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:44.424202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.424906 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.424911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:44.425095 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:44.425737 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:44.425770 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:44.428046 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:44.430877 (+ 2831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:44.430882 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:44.430891 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:44.431229 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:44.431545 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:44.431547 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:44.431549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:44.431692 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:44.432100 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:44.432104 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:44.432355 (+ 251us) client_negotiation.cc:769] Sending connection context
0926 15:15:44.432539 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 15:15:44.432937 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":242,"threads_started":1}
I20260926 15:15:44.435101 13572 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44331
I20260926 15:15:44.435349 13572 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:44.436051 13572 heartbeater.cc:511] Master 127.6.209.254:44331 requested a full tablet report, sending...
I20260926 15:15:44.437393 13117 ts_manager.cc:194] Registered new tserver with Master: ac66294fc8734840a60cbe11e4625260 (127.6.209.195:44185)
I20260926 15:15:44.438239 13117 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.209.195@KRBTEST.COM'} at 127.6.209.195:35607
I20260926 15:15:44.440964 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.209.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.209.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.209.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.253.keytab.
Entry for principal oryx/127.6.209.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.253.keytab.
I20260926 15:15:44.480468 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.253:38809
--webserver_interface=127.6.209.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.253.keytab
--principal=oryx/127.6.209.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:38737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:44331,127.6.209.253:38809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:15:44.541397 13306 heartbeater.cc:503] Master 127.6.209.254:44331 was elected leader, sending a full tablet report...
W20260926 15:15:44.767405 13577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:44.767632 13577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:44.767711 13577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:44.773968 13577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:15:44.774044 13577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:44.774080 13577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:44.774127 13577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:15:44.774156 13577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:15:44.782670 13577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:38737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:44331,127.6.209.253:38809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.253.keytab
--principal=oryx/127.6.209.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.253:38809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.209.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
W20260926 15:15:44.782899 13577 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.6.209.254:44331,127.6.209.253:38809'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 15:15:44.783667 13577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:44.784759 13577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:44.794454 13582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:44.794549 13583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:44.794921 13585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:44.795343 13577 server_base.cc:1034] running on GCE node
I20260926 15:15:44.795949 13577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:44.797120 13577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:44.798321 13577 hybrid_clock.cc:648] HybridClock initialized: now 1790435744798280 us; error 35 us; skew 500 ppm
I20260926 15:15:44.798588 13577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:44 dist-test-slave-j6q7 krb5kdc[13072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435744, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:44.801250 13577 init.cc:377] Logged in from keytab as oryx/127.6.209.253@KRBTEST.COM (short username oryx)
I20260926 15:15:44.802525 13577 webserver.cc:460] Webserver started at http://127.6.209.253:34235/ using document root <none> and password file <none>
I20260926 15:15:44.803139 13577 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:44.803265 13577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:44.803563 13577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:44.806041 13577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/data/instance:
uuid: "cd14dff0bba0414d97637969e5be9c5c"
format_stamp: "Formatted at 2026-09-26 15:15:44 on dist-test-slave-j6q7"
server_key: "8df2ce68b008dca0ada1232092ed0b0b"
server_key_iv: "a125bb9eca7b4e813295bf1f523641ee"
server_key_version: "encryptionkey@0"
I20260926 15:15:44.806679 13577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/wal/instance:
uuid: "cd14dff0bba0414d97637969e5be9c5c"
format_stamp: "Formatted at 2026-09-26 15:15:44 on dist-test-slave-j6q7"
server_key: "8df2ce68b008dca0ada1232092ed0b0b"
server_key_iv: "a125bb9eca7b4e813295bf1f523641ee"
server_key_version: "encryptionkey@0"
I20260926 15:15:44.811233 13577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 15:15:44.813937 13592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:44.814985 13577 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 15:15:44.815155 13577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/wal
uuid: "cd14dff0bba0414d97637969e5be9c5c"
format_stamp: "Formatted at 2026-09-26 15:15:44 on dist-test-slave-j6q7"
server_key: "8df2ce68b008dca0ada1232092ed0b0b"
server_key_iv: "a125bb9eca7b4e813295bf1f523641ee"
server_key_version: "encryptionkey@0"
I20260926 15:15:44.815335 13577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:44.826689 13577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:44.830438 13577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:44.830672 13577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:44.846120 13577 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.253:38809
I20260926 15:15:44.846136 13643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.253:38809 every 8 connection(s)
I20260926 15:15:44.847187 13577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/data/info.pb
I20260926 15:15:44.850395 13644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:44.855073 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 13577
I20260926 15:15:44.855995 13644 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:15:44 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435744, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.253@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM
Sep 26 15:15:44 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.209.253@KRBTEST.COM
I20260926 15:15:44.872879 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:44.858466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36110 (local address 127.6.209.254:44331)
0926 15:15:44.858654 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:44.858660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:44.859286 (+ 626us) server_negotiation.cc:407] Connection header received
0926 15:15:44.859907 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:44.859910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:44.859974 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:44.860045 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:44.861251 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.861966 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.863585 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.864081 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.864155 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:44.867139 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:44.867145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:44.867150 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:44.867184 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:44.870067 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:44.870839 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:44.870842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:44.870844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:44.870938 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:44.871338 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:44.871340 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:44.871342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:44.871698 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:44.871828 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:44.872462 (+ 634us) server_negotiation.cc:299] Negotiation successful
0926 15:15:44.872614 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 15:15:44.874046 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:44.857007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34772 (local address 127.6.209.253:38809)
0926 15:15:44.857695 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:44.857704 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:44.857737 (+ 33us) server_negotiation.cc:407] Connection header received
0926 15:15:44.859701 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:44.859722 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:44.859964 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:44.860130 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:44.861421 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.862783 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.863985 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.864570 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.864679 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:44.868618 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:44.868624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:44.868633 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:44.868654 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:44.871349 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:44.871975 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:44.871981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:44.871983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:44.872138 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:44.872488 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:44.872492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:44.872495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:44.872855 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:44.873118 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:44.873376 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 15:15:44.873572 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":500,"spinlock_wait_cycles":12672,"thread_start_us":309,"threads_started":1}
I20260926 15:15:44.874085 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:44.858460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.0.0.1:36110)
0926 15:15:44.859153 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:44.859189 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:44.859701 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:44.860114 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:44.860119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:44.860719 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:44.861107 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:44.861122 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.862112 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.862116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:44.863471 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:44.863480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.864254 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.864260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:44.864451 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:44.865122 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:44.865147 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:44.866960 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:44.870171 (+ 3211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:44.870179 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:44.870189 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:44.870736 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:44.871033 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:44.871037 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:44.871040 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:44.871248 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:44.871802 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:44.871806 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:44.872087 (+ 281us) client_negotiation.cc:769] Sending connection context
0926 15:15:44.872333 (+ 246us) client_negotiation.cc:240] Negotiation successful
0926 15:15:44.872710 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":54,"thread_start_us":345,"threads_started":1}
I20260926 15:15:44.876300 13644 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:15:44 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435744, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.253@KRBTEST.COM for oryx/127.6.209.253@KRBTEST.COM
I20260926 15:15:44.886466 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:44.877024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:38809 (local address 127.0.0.1:34778)
0926 15:15:44.877147 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:44.877159 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:44.877261 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:44.877523 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:44.877527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:44.877840 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:44.877943 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:44.877950 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.878792 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.878797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:44.879519 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:44.879528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.880123 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.880128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:44.880321 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:44.880725 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:44.880739 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:44.882451 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:44.884797 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:44.884800 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:44.884802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:44.885154 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:44.885423 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:44.885426 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:44.885428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:44.885535 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:44.885908 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:44.885910 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:44.886008 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 15:15:44.886088 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 15:15:44.886259 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":37}
I20260926 15:15:44.886662 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:44.876908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34778 (local address 127.6.209.253:38809)
0926 15:15:44.877037 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:44.877041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:44.877217 (+ 176us) server_negotiation.cc:407] Connection header received
0926 15:15:44.877378 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:44.877382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:44.877423 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:44.877509 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:44.878061 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.878667 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.879634 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.879978 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.880117 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:44.882625 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:44.882629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:44.882632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:44.882656 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:44.884695 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:44.885238 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:44.885241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:44.885243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:44.885342 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:44.885614 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:44.885617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:44.885619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:44.885827 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:44.885913 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:44.886183 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 15:15:44.886358 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":46}
I20260926 15:15:44.890390 13644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c: Bootstrap starting.
I20260926 15:15:44.893033 13644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:44.893900 13644 log.cc:824] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:44.895985 13644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c: No bootstrap required, opened a new log
I20260926 15:15:44.902271 13644 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } }
I20260926 15:15:44.902554 13644 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:44.902637 13644 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd14dff0bba0414d97637969e5be9c5c, State: Initialized, Role: FOLLOWER
I20260926 15:15:44.903111 13644 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } }
I20260926 15:15:44.903754 13650 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } } }
I20260926 15:15:44.903980 13650 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:15:44.904498 13644 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:15:44.913154 13577 master_runner.cc:428] Detected that this master cd14dff0bba0414d97637969e5be9c5c is joining an existing cluster
I20260926 15:15:44.913242 13577 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.209.253:38809
W20260926 15:15:44.915086 13661 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:15:44.915205 13661 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
WARNING: no policy specified for oryx/127.6.209.252@KRBTEST.COM; defaulting to no policy
I20260926 15:15:44.951797 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:44.917009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36122 (local address 127.6.209.254:44331)
0926 15:15:44.917218 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:44.917227 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:44.927574 (+ 10347us) server_negotiation.cc:407] Connection header received
0926 15:15:44.935928 (+ 8354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:44.935932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:44.936007 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:44.936069 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:44.937435 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.938563 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.940620 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.941333 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.941447 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:44.944182 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:44.944189 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:44.944195 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:44.944230 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:44.947651 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:44.949097 (+ 1446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:44.949101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:44.949105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:44.949265 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:44.949662 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:44.949668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:44.949672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:44.949986 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:44.950104 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:44.950975 (+ 871us) server_negotiation.cc:299] Negotiation successful
0926 15:15:44.951181 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 15:15:44.953472 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:44.916591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.0.0.1:36122)
0926 15:15:44.927483 (+ 10892us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:44.927502 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:44.927665 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:44.936256 (+ 8591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:44.936261 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:44.936882 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:44.937267 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:44.937280 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.938718 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.938723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:44.940437 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:44.940450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.941720 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.941726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:44.941906 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:44.942558 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:44.942577 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:44.943991 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:44.947863 (+ 3872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:44.947870 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:44.947875 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:44.948413 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:44.949401 (+ 988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:44.949407 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:44.949410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:44.949548 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:44.950179 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:44.950183 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:44.950643 (+ 460us) client_negotiation.cc:769] Sending connection context
0926 15:15:44.952584 (+ 1941us) client_negotiation.cc:240] Negotiation successful
0926 15:15:44.953002 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10752,"thread_start_us":12448,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.209.252@KRBTEST.COM" created.
I20260926 15:15:44.963681 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:44.916626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34780 (local address 127.6.209.253:38809)
0926 15:15:44.916794 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:44.916801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:44.938863 (+ 22062us) server_negotiation.cc:407] Connection header received
0926 15:15:44.938921 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:44.938928 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:44.939004 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:44.939103 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:44.940460 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.941674 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.943877 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.944620 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.944706 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:44.949822 (+ 5116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:44.949828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:44.949833 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:44.949861 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:44.953422 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:44.961418 (+ 7996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:44.961425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:44.961429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:44.961570 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:44.961978 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:44.961985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:44.961990 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:44.962388 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:44.962500 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:44.963037 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 15:15:44.963253 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":51}
I20260926 15:15:44.964694 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:44.916765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:38809 (local address 127.0.0.1:34780)
0926 15:15:44.928797 (+ 12032us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:44.928839 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:44.928955 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:44.939198 (+ 10243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:44.939205 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:44.939874 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:44.940319 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:44.940334 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.941934 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.941942 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:44.943703 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:44.943722 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.945033 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.945040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:44.945228 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:44.946976 (+ 1748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:44.947288 (+ 312us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:44.948962 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:44.960686 (+ 11724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:44.960690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:44.960693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:44.961306 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:44.961703 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:44.961707 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:44.961710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:44.961872 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:44.962568 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:44.962574 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:44.962796 (+ 222us) client_negotiation.cc:769] Sending connection context
0926 15:15:44.963888 (+ 1092us) client_negotiation.cc:240] Negotiation successful
0926 15:15:44.964170 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11915,"mutex_wait_us":27,"thread_start_us":18861,"threads_started":1}
Sep 26 15:15:44 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435742, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.254@KRBTEST.COM for oryx/127.6.209.253@KRBTEST.COM
I20260926 15:15:44.975639 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:44.959389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:38809 (local address 127.0.0.1:34794)
0926 15:15:44.960250 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:44.960280 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:44.960476 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:44.961285 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:44.961292 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:44.962046 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:44.962337 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:44.962348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.963942 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.963947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:44.965333 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:44.965347 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.966330 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.966338 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:44.966546 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:44.967313 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:44.967340 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:44.970035 (+ 2695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:44.973502 (+ 3467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:44.973509 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:44.973513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:44.973821 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:44.974154 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:44.974156 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:44.974158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:44.974256 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:44.974743 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:44.974747 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:44.974885 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 15:15:44.975049 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 15:15:44.975365 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":591,"threads_started":1}
I20260926 15:15:44.975683 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:44.959407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34794 (local address 127.6.209.253:38809)
0926 15:15:44.959905 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:44.959909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:44.960370 (+ 461us) server_negotiation.cc:407] Connection header received
0926 15:15:44.960651 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:44.960658 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:44.960737 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:44.960891 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:44.962518 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.963789 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.965488 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:44.966173 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:44.966291 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:44.970239 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:44.970245 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:44.970249 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:44.970289 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:44.973380 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:44.973922 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:44.973927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:44.973929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:44.974061 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:44.974347 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:44.974351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:44.974353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:44.974629 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:44.974751 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:44.975126 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 15:15:44.975311 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":374,"thread_start_us":250,"threads_started":1}
I20260926 15:15:44.977385 13117 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.209.253:38809: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c"
member_type: NON_VOTER
last_known_addr {
host: "127.6.209.253"
port: 38809
}
attrs {
promote: true
}
}
dest_uuid: "487b6fe6e7264e4db494c8057789998b"
cas_config_opid_index: -1
I20260926 15:15:44.978420 13117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: NON_VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.209.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.252.keytab.
Entry for principal oryx/127.6.209.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.252.keytab.
I20260926 15:15:44.980734 13439 heartbeater.cc:503] Master 127.6.209.254:44331 was elected leader, sending a full tablet report...
I20260926 15:15:44.982131 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.252:40519
--webserver_interface=127.6.209.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.252.keytab
--principal=oryx/127.6.209.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:38737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:44331,127.6.209.253:38809,127.6.209.252:40519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:15:44.989709 13675 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER cd14dff0bba0414d97637969e5be9c5c (127.6.209.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: NON_VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: true } } }
I20260926 15:15:44.990653 13619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:44.993765 13619 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [term 1 FOLLOWER]: Refusing update from remote peer 487b6fe6e7264e4db494c8057789998b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 15:15:44.994620 13674 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: NON_VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:45.006781 13675 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.209.253:38809
I20260926 15:15:45.007413 13577 master.cc:537] Master@127.6.209.253:38809 shutting down...
I20260926 15:15:45.012583 13674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "487b6fe6e7264e4db494c8057789998b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: NON_VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: true } } }
I20260926 15:15:45.012825 13674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:45.028124 13619 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER cd14dff0bba0414d97637969e5be9c5c (127.6.209.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: NON_VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: true } } }
I20260926 15:15:45.056048 13685 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "487b6fe6e7264e4db494c8057789998b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: NON_VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: true } } }
I20260926 15:15:45.056564 13685 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [sys.catalog]: This master's current role is: LEARNER
I20260926 15:15:45.056500 13650 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 487b6fe6e7264e4db494c8057789998b. Latest consensus state: current_term: 1 leader_uuid: "487b6fe6e7264e4db494c8057789998b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: NON_VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: true } } }
I20260926 15:15:45.056689 13650 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [sys.catalog]: This master's current role is: LEARNER
I20260926 15:15:45.057031 13681 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "487b6fe6e7264e4db494c8057789998b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: NON_VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: true } } }
I20260926 15:15:45.057389 13681 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [sys.catalog]: This master's current role is: LEARNER
I20260926 15:15:45.059465 13577 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [term 1 LEARNER]: Raft consensus shutting down.
I20260926 15:15:45.059803 13577 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [term 1 LEARNER]: Raft consensus is shut down!
I20260926 15:15:45.059945 13577 tablet_replica.cc:330] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c: stopping tablet replica
I20260926 15:15:45.066140 13577 master.cc:552] Master@127.6.209.253:38809 shutdown complete.
I20260926 15:15:45.066221 13577 master_runner.cc:305] Clearing existing system tablet
I20260926 15:15:45.066536 13577 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:45.066625 13577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:45.068799 13577 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:15:45.070071 13687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:45.070402 13577 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:45.070667 13577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:15:45.070745 13577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/wal
uuid: "cd14dff0bba0414d97637969e5be9c5c"
format_stamp: "Formatted at 2026-09-26 15:15:44 on dist-test-slave-j6q7"
server_key: "8df2ce68b008dca0ada1232092ed0b0b"
server_key_iv: "a125bb9eca7b4e813295bf1f523641ee"
server_key_version: "encryptionkey@0"
I20260926 15:15:45.073222 13577 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:45.076882 13577 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 15:15:45.077090 13577 log.cc:1196] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 15:15:45.077474 13577 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c: Deleting consensus metadata
I20260926 15:15:45.077720 13577 master_runner.cc:315] Copying system tablet from 127.6.209.254:44331
I20260926 15:15:45.079989 13577 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.209.254:44331
I20260926 15:15:45.095074 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:45.080642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36124 (local address 127.6.209.254:44331)
0926 15:15:45.080926 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:45.080936 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:45.085344 (+ 4408us) server_negotiation.cc:407] Connection header received
0926 15:15:45.085528 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:45.085533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:45.085601 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:45.085690 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:45.086396 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.087061 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.088372 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.088763 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.088917 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:45.090429 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:45.090434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:45.090438 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:45.090473 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:45.092902 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:45.093426 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:45.093433 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:45.093436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:45.093564 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:45.093883 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:45.093887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:45.093890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:45.094107 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:45.094249 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:45.094584 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 15:15:45.094754 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:15:45.095095 13693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:45.080628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.0.0.1:36124)
0926 15:15:45.085243 (+ 4615us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:45.085269 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:45.085404 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:45.085711 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:45.085714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:45.086073 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:45.086255 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:45.086264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.087194 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.087199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:45.088257 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:45.088268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.088932 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.088937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:45.089097 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:45.089643 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:45.089657 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:45.090279 (+ 622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:45.093022 (+ 2743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:45.093028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:45.093031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:45.093329 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:45.093657 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:45.093662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:45.093665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:45.093786 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:45.094209 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:45.094212 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:45.094362 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 15:15:45.094492 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 15:15:45.094733 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4519,"thread_start_us":4420,"threads_started":1}
I20260926 15:15:45.096091 13137 tablet_copy_service.cc:140] P 487b6fe6e7264e4db494c8057789998b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer cd14dff0bba0414d97637969e5be9c5c ({username='oryx', principal='oryx/127.6.209.253@KRBTEST.COM'} at 127.0.0.1:36124)
I20260926 15:15:45.096323 13137 tablet_copy_service.cc:161] P 487b6fe6e7264e4db494c8057789998b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer cd14dff0bba0414d97637969e5be9c5c at {username='oryx', principal='oryx/127.6.209.253@KRBTEST.COM'} at 127.0.0.1:36124: session id = cd14dff0bba0414d97637969e5be9c5c-00000000000000000000000000000000
I20260926 15:15:45.099294 13137 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 15:15:45.100739 13577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 15:15:45.104784 13577 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c: tablet copy: Starting download of 0 data blocks...
I20260926 15:15:45.105072 13577 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c: tablet copy: Starting download of 1 WAL segments...
I20260926 15:15:45.107199 13577 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 15:15:45.110539 13137 tablet_copy_service.cc:342] P 487b6fe6e7264e4db494c8057789998b: Request end of tablet copy session cd14dff0bba0414d97637969e5be9c5c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.209.253@KRBTEST.COM'} at 127.0.0.1:36124
I20260926 15:15:45.110700 13137 tablet_copy_service.cc:433] P 487b6fe6e7264e4db494c8057789998b: ending tablet copy session cd14dff0bba0414d97637969e5be9c5c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer cd14dff0bba0414d97637969e5be9c5c
I20260926 15:15:45.119535 13577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:45.121804 13694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:45.125969 13695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:45.133417 13697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:45.133915 13577 server_base.cc:1034] running on GCE node
I20260926 15:15:45.134234 13577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:45.134780 13577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:45.136018 13577 hybrid_clock.cc:648] HybridClock initialized: now 1790435745135937 us; error 59 us; skew 500 ppm
I20260926 15:15:45.136250 13577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:45 dist-test-slave-j6q7 krb5kdc[13072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435745, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:45.138345 13577 init.cc:377] Logged in from keytab as oryx/127.6.209.253@KRBTEST.COM (short username oryx)
I20260926 15:15:45.141651 13577 webserver.cc:460] Webserver started at http://127.6.209.253:33543/ using document root <none> and password file <none>
I20260926 15:15:45.141963 13577 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:45.142068 13577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:45.144098 13577 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 15:15:45.145588 13704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:45.145924 13577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:15:45.146042 13577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/wal
uuid: "cd14dff0bba0414d97637969e5be9c5c"
format_stamp: "Formatted at 2026-09-26 15:15:44 on dist-test-slave-j6q7"
server_key: "8df2ce68b008dca0ada1232092ed0b0b"
server_key_iv: "a125bb9eca7b4e813295bf1f523641ee"
server_key_version: "encryptionkey@0"
I20260926 15:15:45.146185 13577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:45.159737 13577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:45.162644 13577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:45.189117 13577 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.253:38809
I20260926 15:15:45.189249 13755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.253:38809 every 8 connection(s)
I20260926 15:15:45.189850 13577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-1/data/info.pb
I20260926 15:15:45.196095 13756 sys_catalog.cc:263] Verifying existing consensus state
I20260926 15:15:45.201977 13756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c: Bootstrap starting.
I20260926 15:15:45.213786 13756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:15:45.214186 13756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c: Bootstrap complete.
I20260926 15:15:45.214851 13756 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: NON_VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: true } }
I20260926 15:15:45.215035 13756 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cd14dff0bba0414d97637969e5be9c5c, State: Initialized, Role: LEARNER
I20260926 15:15:45.215224 13756 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: NON_VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: true } }
I20260926 15:15:45.215876 13759 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: NON_VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: true } } }
I20260926 15:15:45.216142 13756 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:15:45.216236 13759 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [sys.catalog]: This master's current role is: LEARNER
I20260926 15:15:45.229131 13770 catalog_manager.cc:1259] Loaded cluster ID: aaebb9b0e62d4ad5b164b4119414725a
I20260926 15:15:45.229223 13770 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c: loading cluster ID for follower catalog manager: success
I20260926 15:15:45.231510 13770 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c: acquiring CA information for follower catalog manager: success
I20260926 15:15:45.233143 13770 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 15:15:45.377313 13676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:45.377554 13676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:45.378057 13676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:45.384106 13676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:15:45.384341 13676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:45.384459 13676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:45.384541 13676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:15:45.384616 13676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:15:45.393324 13676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:38737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:44331,127.6.209.253:38809,127.6.209.252:40519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.252.keytab
--principal=oryx/127.6.209.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.252:40519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.209.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:45.394348 13676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:45.395418 13676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:45.407447 13775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:45.407560 13776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:45.408517 13778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:45.409185 13676 server_base.cc:1034] running on GCE node
I20260926 15:15:45.409803 13676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:45.411003 13676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:45.412204 13676 hybrid_clock.cc:648] HybridClock initialized: now 1790435745411973 us; error 225 us; skew 500 ppm
I20260926 15:15:45.412550 13676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:45 dist-test-slave-j6q7 krb5kdc[13072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435745, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:45.415819 13676 init.cc:377] Logged in from keytab as oryx/127.6.209.252@KRBTEST.COM (short username oryx)
I20260926 15:15:45.420295 13676 webserver.cc:460] Webserver started at http://127.6.209.252:37283/ using document root <none> and password file <none>
I20260926 15:15:45.421588 13676 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:45.421710 13676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:45.422025 13676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:45.425199 13676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/data/instance:
uuid: "47996a5d983a4193afbd0f1ee829e0f5"
format_stamp: "Formatted at 2026-09-26 15:15:45 on dist-test-slave-j6q7"
server_key: "b0bdda558ddec9b3de57c0a1e143cfc5"
server_key_iv: "37a63f8bd0a81d47b71d1b86aa4c1ed2"
server_key_version: "encryptionkey@0"
I20260926 15:15:45.426026 13676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/wal/instance:
uuid: "47996a5d983a4193afbd0f1ee829e0f5"
format_stamp: "Formatted at 2026-09-26 15:15:45 on dist-test-slave-j6q7"
server_key: "b0bdda558ddec9b3de57c0a1e143cfc5"
server_key_iv: "37a63f8bd0a81d47b71d1b86aa4c1ed2"
server_key_version: "encryptionkey@0"
I20260926 15:15:45.431329 13676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 15:15:45.434654 13785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:45.435818 13676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:15:45.435938 13676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/wal
uuid: "47996a5d983a4193afbd0f1ee829e0f5"
format_stamp: "Formatted at 2026-09-26 15:15:45 on dist-test-slave-j6q7"
server_key: "b0bdda558ddec9b3de57c0a1e143cfc5"
server_key_iv: "37a63f8bd0a81d47b71d1b86aa4c1ed2"
server_key_version: "encryptionkey@0"
I20260926 15:15:45.436065 13676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:45.440435 13572 heartbeater.cc:503] Master 127.6.209.254:44331 was elected leader, sending a full tablet report...
I20260926 15:15:45.454722 13676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:45.457934 13676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:45.458199 13676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:45.474855 13676 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.252:40519
I20260926 15:15:45.474882 13836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.252:40519 every 8 connection(s)
I20260926 15:15:45.475916 13676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/data/info.pb
I20260926 15:15:45.478924 13837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:45.483892 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 13676
I20260926 15:15:45.484786 13837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:15:45 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435745, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.252@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM
Sep 26 15:15:45 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.209.252@KRBTEST.COM
I20260926 15:15:45.503667 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:45.487839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36128 (local address 127.6.209.254:44331)
0926 15:15:45.488014 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:45.488020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:45.488751 (+ 731us) server_negotiation.cc:407] Connection header received
0926 15:15:45.489134 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:45.489138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:45.489210 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:45.489299 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:45.490812 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.491778 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.493471 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.494043 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.494147 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:45.497981 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:45.497985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:45.497988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:45.498040 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:45.500784 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:45.501673 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:45.501677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:45.501679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:45.501796 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:45.502242 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:45.502246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:45.502249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:45.502565 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:45.502710 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:45.503340 (+ 630us) server_negotiation.cc:299] Negotiation successful
0926 15:15:45.503474 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 15:15:45.504674 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:45.487738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.0.0.1:36128)
0926 15:15:45.488611 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:45.488649 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:45.488940 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:45.489430 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:45.489436 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:45.490207 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:45.490644 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:45.490665 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.491891 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.491895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:45.493358 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:45.493371 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.494201 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.494205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:45.494329 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:45.495132 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:45.495166 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:45.497801 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:45.500968 (+ 3167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:45.500979 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:45.500990 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:45.501566 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:45.501901 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:45.501906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:45.501911 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:45.502126 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:45.502676 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:45.502681 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:45.502992 (+ 311us) client_negotiation.cc:769] Sending connection context
0926 15:15:45.503215 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 15:15:45.503607 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"mutex_wait_us":35,"thread_start_us":515,"threads_started":1}
I20260926 15:15:45.505215 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:45.485943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57096 (local address 127.6.209.252:40519)
0926 15:15:45.486519 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:45.486527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:45.486558 (+ 31us) server_negotiation.cc:407] Connection header received
0926 15:15:45.488855 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:45.488888 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:45.489105 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:45.489287 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:45.490903 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.492166 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.493597 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.494299 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.494372 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:45.498951 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:45.498958 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:45.498967 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:45.498993 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:45.502120 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:45.502800 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:45.502806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:45.502809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:45.502970 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:45.503287 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:45.503291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:45.503293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:45.503621 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:45.503877 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:45.504399 (+ 522us) server_negotiation.cc:299] Negotiation successful
0926 15:15:45.504587 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":430,"spinlock_wait_cycles":15104,"thread_start_us":275,"threads_started":1}
I20260926 15:15:45.506050 13837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:15:45 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435745, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.252@KRBTEST.COM for oryx/127.6.209.253@KRBTEST.COM
I20260926 15:15:45.517738 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:45.506686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:38809 (local address 127.0.0.1:34796)
0926 15:15:45.506832 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:45.506849 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:45.506940 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:45.507746 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:45.507749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:45.508019 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:45.508123 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:45.508130 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.509545 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.509555 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:45.510327 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:45.510335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.510852 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.510856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:45.510964 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:45.511355 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:45.511368 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:45.513067 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:45.515942 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:45.515945 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:45.515947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:45.516221 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:45.516504 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:45.516507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:45.516509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:45.516601 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:45.517143 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:45.517146 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:45.517247 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 15:15:45.517359 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 15:15:45.517527 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 15:15:45.517994 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:45.506759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34796 (local address 127.6.209.253:38809)
0926 15:15:45.507493 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:45.507500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:45.507518 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:15:45.507574 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:45.507577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:45.507629 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:45.507710 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:45.508272 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.509364 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.510420 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.510731 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.510813 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:45.513363 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:45.513369 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:45.513373 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:45.513402 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:45.515840 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:45.516309 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:45.516312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:45.516313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:45.516409 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:45.516680 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:45.516683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:45.516684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:45.517025 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:45.517188 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:45.517438 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 15:15:45.517568 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":461,"threads_started":1}
I20260926 15:15:45.518884 13837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.252" port: 40519 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:15:45 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435745, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.252@KRBTEST.COM for oryx/127.6.209.252@KRBTEST.COM
I20260926 15:15:45.529320 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:45.519566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:40519 (local address 127.0.0.1:57102)
0926 15:15:45.519675 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:45.519689 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:45.519784 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:45.520021 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:45.520024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:45.520333 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:45.520437 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:45.520443 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.521404 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.521411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:45.522161 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:45.522169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.522741 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.522745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:45.522974 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:45.523357 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:45.523369 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:45.525068 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:45.527265 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:45.527270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:45.527274 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:45.527615 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:45.527930 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:45.527937 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:45.527940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:45.528074 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:45.528489 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:45.528492 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:45.528593 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 15:15:45.528733 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 15:15:45.528969 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":36}
I20260926 15:15:45.529352 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:45.519494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57102 (local address 127.6.209.252:40519)
0926 15:15:45.519640 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:45.519645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:45.519745 (+ 100us) server_negotiation.cc:407] Connection header received
0926 15:15:45.519877 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:45.519881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:45.519931 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:45.520010 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:45.520548 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.521282 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.522284 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.522623 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.522734 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:45.525211 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:45.525216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:45.525219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:45.525239 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:45.527151 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:45.527728 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:45.527735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:45.527743 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:45.527834 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:45.528181 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:45.528187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:45.528189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:45.528398 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:45.528515 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:45.528796 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 15:15:45.529064 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":39}
I20260926 15:15:45.533604 13837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5: Bootstrap starting.
I20260926 15:15:45.536230 13837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:45.537160 13837 log.cc:824] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:45.539505 13837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5: No bootstrap required, opened a new log
I20260926 15:15:45.546226 13837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 40519 } }
I20260926 15:15:45.546484 13837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:45.546558 13837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47996a5d983a4193afbd0f1ee829e0f5, State: Initialized, Role: FOLLOWER
I20260926 15:15:45.547049 13837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 40519 } }
I20260926 15:15:45.547981 13844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 40519 } } }
I20260926 15:15:45.548329 13844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:15:45.548945 13837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 15:15:45.556103 13855 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:15:45.556218 13855 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:15:45.559110 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 13178
I20260926 15:15:45.560243 13676 master_runner.cc:428] Detected that this master 47996a5d983a4193afbd0f1ee829e0f5 is joining an existing cluster
I20260926 15:15:45.560343 13676 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.209.252:40519
Sep 26 15:15:45 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.209.193@KRBTEST.COM
I20260926 15:15:45.577117 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:45.559951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46802 (local address 127.6.209.193:38531)
0926 15:15:45.561292 (+ 1341us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:45.561301 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:45.561328 (+ 27us) server_negotiation.cc:407] Connection header received
0926 15:15:45.561390 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:45.561397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:45.561617 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:45.561824 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:45.562866 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.563781 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.564824 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.565315 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.565428 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:45.569483 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:45.569491 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:45.569500 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:45.569525 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:45.572768 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:45.573696 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:45.573706 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:45.573711 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:45.573864 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:45.574345 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:45.574352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:45.574357 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:45.574773 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:45.576192 (+ 1419us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:45.576364 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 15:15:45.576578 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1190,"thread_start_us":1053,"threads_started":1}
I20260926 15:15:45.578738 13301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:45.605525 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:45.571325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.0.0.1:36144)
0926 15:15:45.581828 (+ 10503us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:45.581846 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:45.581966 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:45.588941 (+ 6975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:45.588946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:45.589574 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:45.589926 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:45.589937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.591548 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.591554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:45.593119 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:45.593132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.594595 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.594601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:45.594770 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:45.595474 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:45.595499 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:45.596590 (+ 1091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:45.602407 (+ 5817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:45.602413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:45.602419 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:45.602881 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:45.603417 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:45.603422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:45.603426 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:45.603623 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:45.604334 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:45.604341 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:45.604530 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 15:15:45.604696 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 15:15:45.604989 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9803,"thread_start_us":23794,"threads_started":1}
I20260926 15:15:45.606424 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:45.570849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36144 (local address 127.6.209.254:44331)
0926 15:15:45.571016 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:45.571021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:45.581918 (+ 10897us) server_negotiation.cc:407] Connection header received
0926 15:15:45.582071 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:45.582077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:45.582165 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:45.582256 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:45.590111 (+ 7855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.591047 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.593283 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.594266 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.594383 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:45.596865 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:45.596871 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:45.596876 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:45.596913 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:45.600030 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:45.603041 (+ 3011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:45.603048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:45.603052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:45.603273 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:45.603774 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:45.603781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:45.603785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:45.604123 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:45.604275 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:45.605856 (+ 1581us) server_negotiation.cc:299] Negotiation successful
0926 15:15:45.606073 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 15:15:45.617340 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:45.576871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:38809 (local address 127.0.0.1:34806)
0926 15:15:45.595578 (+ 18707us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:45.595599 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:45.595765 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:45.596266 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:45.596271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:45.596936 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:45.597279 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:45.597295 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.598689 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.598694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:45.600906 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:45.600923 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.602607 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.602613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:45.602795 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:45.603859 (+ 1064us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:45.603884 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:45.604849 (+ 965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:45.612455 (+ 7606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:45.612459 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:45.612462 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:45.612920 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:45.613862 (+ 942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:45.613866 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:45.613869 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:45.614027 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:45.615676 (+ 1649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:45.615680 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:45.615889 (+ 209us) client_negotiation.cc:769] Sending connection context
0926 15:15:45.616478 (+ 589us) client_negotiation.cc:240] Negotiation successful
0926 15:15:45.616781 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18586,"mutex_wait_us":380,"spinlock_wait_cycles":24448,"thread_start_us":1796,"threads_started":1}
I20260926 15:15:45.619115 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:45.571035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:40519 (local address 127.0.0.1:57112)
0926 15:15:45.578951 (+ 7916us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:45.578972 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:45.579592 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:45.580226 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:45.580232 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:45.581057 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:45.597780 (+ 16723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:45.597793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.599079 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.599084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:45.600765 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:45.600779 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.603306 (+ 2527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.603315 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:45.603507 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:45.605076 (+ 1569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:45.605107 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:45.606006 (+ 899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:45.613322 (+ 7316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:45.613326 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:45.613329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:45.613728 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:45.614864 (+ 1136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:45.614870 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:45.614873 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:45.615018 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:45.618131 (+ 3113us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:45.618137 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:45.618310 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 15:15:45.618426 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 15:15:45.618672 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7778,"mutex_wait_us":6,"thread_start_us":9962,"threads_started":1}
I20260926 15:15:45.620169 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:45.570962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57112 (local address 127.6.209.252:40519)
0926 15:15:45.571151 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:45.571157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:45.579050 (+ 7893us) server_negotiation.cc:407] Connection header received
0926 15:15:45.579851 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:45.579858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:45.579928 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:45.579994 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:45.597939 (+ 17945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.598871 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.600982 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.602104 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.602201 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:45.606186 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:45.606191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:45.606196 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:45.606226 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:45.611370 (+ 5144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:45.614155 (+ 2785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:45.614159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:45.614162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:45.614296 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:45.615135 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:45.615139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:45.615142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:45.615510 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:45.619542 (+ 4032us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:45.619661 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 15:15:45.619855 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 15:15:45.620901 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:45.576387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34806 (local address 127.6.209.253:38809)
0926 15:15:45.576531 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:45.576536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:45.595685 (+ 19149us) server_negotiation.cc:407] Connection header received
0926 15:15:45.595907 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:45.595915 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:45.595997 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:45.596065 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:45.597469 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.598516 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.601125 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.602194 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.602286 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:45.606811 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:45.606816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:45.606822 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:45.606859 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:45.612314 (+ 5455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:45.613061 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:45.613066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:45.613069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:45.613192 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:45.614427 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:45.614431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:45.614435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:45.614740 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:45.617786 (+ 3046us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:45.617926 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 15:15:45.620535 (+ 2609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 15:15:45 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435742, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.254@KRBTEST.COM for oryx/127.6.209.252@KRBTEST.COM
I20260926 15:15:45.653306 13866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "487b6fe6e7264e4db494c8057789998b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: NON_VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: true } } }
I20260926 15:15:45.653569 13866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:45.669108 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:45.612896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57124 (local address 127.6.209.252:40519)
0926 15:15:45.622443 (+ 9547us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:45.622450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:45.640387 (+ 17937us) server_negotiation.cc:407] Connection header received
0926 15:15:45.640635 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:45.640641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:45.640752 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:45.640818 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:45.642008 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.643007 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.644846 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.645561 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.645653 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:45.660592 (+ 14939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:45.660599 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:45.660605 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:45.660645 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:45.664177 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:45.664910 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:45.664916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:45.664921 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:45.665039 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:45.666917 (+ 1878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:45.666922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:45.666925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:45.667680 (+ 755us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:45.667842 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:45.668507 (+ 665us) server_negotiation.cc:299] Negotiation successful
0926 15:15:45.668788 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9442,"thread_start_us":27019,"threads_started":1}
I20260926 15:15:45.669998 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:45.612550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:40519 (local address 127.0.0.1:57124)
0926 15:15:45.640302 (+ 27752us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:45.640322 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:45.640474 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:45.641127 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:45.641133 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:45.641591 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:45.641792 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:45.641804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.643184 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.643189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:45.644500 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:45.644514 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.645868 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.645925 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:45.646094 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:45.646799 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:45.646817 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:45.650034 (+ 3217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:45.664294 (+ 14260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:45.664297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:45.664301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:45.664769 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:45.665209 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:45.665215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:45.665219 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:45.666697 (+ 1478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:45.667949 (+ 1252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:45.667954 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:45.668154 (+ 200us) client_negotiation.cc:769] Sending connection context
0926 15:15:45.669308 (+ 1154us) client_negotiation.cc:240] Negotiation successful
0926 15:15:45.669535 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27638,"thread_start_us":32037,"threads_started":1}
I20260926 15:15:45.670686 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:45.647369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34808 (local address 127.6.209.253:38809)
0926 15:15:45.647524 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:45.647530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:45.661992 (+ 14462us) server_negotiation.cc:407] Connection header received
0926 15:15:45.662240 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:45.662246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:45.662332 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:45.662404 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:45.663199 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.664105 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.668318 (+ 4213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.669427 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.669519 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:45.669700 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:45.670312 (+ 612us) server_negotiation.cc:299] Negotiation successful
0926 15:15:45.670433 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 15:15:45.671350 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:45.646911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:38809 (local address 127.0.0.1:34808)
0926 15:15:45.661892 (+ 14981us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:45.661922 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:45.662084 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:45.662572 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:45.662620 (+ 48us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:45.662656 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:45.663026 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:45.663038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.664271 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.664276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:45.668147 (+ 3871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:45.668158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.669970 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.669978 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:45.670127 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:45.670191 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 15:15:45.670862 (+ 671us) client_negotiation.cc:240] Negotiation successful
0926 15:15:45.670975 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13697,"thread_start_us":14493,"threads_started":1}
I20260926 15:15:45.671622 13117 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.209.252:40519: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5"
member_type: NON_VOTER
last_known_addr {
host: "127.6.209.252"
port: 40519
}
attrs {
promote: true
}
}
dest_uuid: "487b6fe6e7264e4db494c8057789998b"
cas_config_opid_index: 5
I20260926 15:15:45.672277 13759 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 487b6fe6e7264e4db494c8057789998b. Latest consensus state: current_term: 1 leader_uuid: "487b6fe6e7264e4db494c8057789998b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: NON_VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: true } } }
I20260926 15:15:45.672282 13117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: NON_VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: true } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: NON_VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: true } }
I20260926 15:15:45.672467 13759 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [sys.catalog]: This master's current role is: LEARNER
I20260926 15:15:45.675715 13812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:45.675995 13731 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [term 1 LEARNER]: Refusing update from remote peer 487b6fe6e7264e4db494c8057789998b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 15:15:45.675567 13867 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 47996a5d983a4193afbd0f1ee829e0f5 (127.6.209.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: NON_VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: true } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: NON_VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: true } } }
I20260926 15:15:45.676496 13866 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: NON_VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 15:15:45.679345 13812 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [term 1 FOLLOWER]: Refusing update from remote peer 487b6fe6e7264e4db494c8057789998b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 15:15:45.680219 13867 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.209.252:40519
I20260926 15:15:45.680244 13866 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [LEADER]: Connected to new peer: Peer: permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: NON_VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:45.680785 13676 master.cc:537] Master@127.6.209.252:40519 shutting down...
W20260926 15:15:45.682626 13102 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 15:15:45.696262 13102 consensus_peers.cc:487] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b -> Peer 47996a5d983a4193afbd0f1ee829e0f5 (127.6.209.252:40519): Couldn't send request to peer 47996a5d983a4193afbd0f1ee829e0f5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 15:15:45.696658 13731 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 47996a5d983a4193afbd0f1ee829e0f5 (127.6.209.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: NON_VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: true } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: NON_VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: true } } }
I20260926 15:15:45.700408 13759 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "487b6fe6e7264e4db494c8057789998b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: NON_VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: true } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: NON_VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: true } } }
I20260926 15:15:45.700783 13759 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [sys.catalog]: This master's current role is: LEARNER
I20260926 15:15:45.701624 13867 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b: attempting to promote NON_VOTER cd14dff0bba0414d97637969e5be9c5c to VOTER
I20260926 15:15:45.701869 13866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "487b6fe6e7264e4db494c8057789998b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: NON_VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: true } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: NON_VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: true } } }
I20260926 15:15:45.702100 13866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:45.701644 13870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "487b6fe6e7264e4db494c8057789998b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: NON_VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: true } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: NON_VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: true } } }
I20260926 15:15:45.702677 13870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:45.702625 13867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: NON_VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: true } }
I20260926 15:15:45.704715 13731 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [term 1 LEARNER]: Refusing update from remote peer 487b6fe6e7264e4db494c8057789998b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 15:15:45.705150 13870 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260926 15:15:45.705456 13102 consensus_peers.cc:487] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b -> Peer 47996a5d983a4193afbd0f1ee829e0f5 (127.6.209.252:40519): Couldn't send request to peer 47996a5d983a4193afbd0f1ee829e0f5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 15:15:45.709833 13870 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, cd14dff0bba0414d97637969e5be9c5c (127.6.209.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: NON_VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: true } } }
I20260926 15:15:45.713037 13866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "487b6fe6e7264e4db494c8057789998b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: NON_VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: true } } }
I20260926 15:15:45.713253 13866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:45.713639 13866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "487b6fe6e7264e4db494c8057789998b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: NON_VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: true } } }
I20260926 15:15:45.713828 13866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:45.714715 13730 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, cd14dff0bba0414d97637969e5be9c5c (127.6.209.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: NON_VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: true } } }
W20260926 15:15:45.717059 13791 tablet.cc:2369] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:15:45.720773 13844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:15:45.721045 13844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 40519 } }
I20260926 15:15:45.722496 13844 leader_election.cc:288] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 487b6fe6e7264e4db494c8057789998b (127.6.209.254:44331), cd14dff0bba0414d97637969e5be9c5c (127.6.209.253:38809)
I20260926 15:15:45.722841 13127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47996a5d983a4193afbd0f1ee829e0f5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "487b6fe6e7264e4db494c8057789998b" is_pre_election: true
I20260926 15:15:45.727541 13759 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "487b6fe6e7264e4db494c8057789998b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: NON_VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: true } } }
I20260926 15:15:45.727792 13759 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:15:45.728624 13731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47996a5d983a4193afbd0f1ee829e0f5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd14dff0bba0414d97637969e5be9c5c" is_pre_election: true
I20260926 15:15:45.728709 13676 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:45.729049 13676 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:45.729221 13676 tablet_replica.cc:330] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5: stopping tablet replica
I20260926 15:15:45.729321 13789 leader_election.cc:302] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47996a5d983a4193afbd0f1ee829e0f5; no voters: 487b6fe6e7264e4db494c8057789998b, cd14dff0bba0414d97637969e5be9c5c
W20260926 15:15:45.729609 13789 raft_consensus.cc:2670] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260926 15:15:45.745561 13676 master.cc:552] Master@127.6.209.252:40519 shutdown complete.
I20260926 15:15:45.745679 13676 master_runner.cc:305] Clearing existing system tablet
I20260926 15:15:45.746052 13676 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:45.746160 13676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:45.748981 13676 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260926 15:15:45.750411 13877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:45.750679 13676 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:45.750967 13676 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:15:45.751086 13676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/wal
uuid: "47996a5d983a4193afbd0f1ee829e0f5"
format_stamp: "Formatted at 2026-09-26 15:15:45 on dist-test-slave-j6q7"
server_key: "b0bdda558ddec9b3de57c0a1e143cfc5"
server_key_iv: "37a63f8bd0a81d47b71d1b86aa4c1ed2"
server_key_version: "encryptionkey@0"
I20260926 15:15:45.755538 13676 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:45.762041 13676 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 15:15:45.762212 13676 log.cc:1196] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 15:15:45.762600 13676 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5: Deleting consensus metadata
I20260926 15:15:45.762835 13676 master_runner.cc:315] Copying system tablet from 127.6.209.254:44331
I20260926 15:15:45.767254 13676 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5: tablet copy: Beginning tablet copy session from remote peer at address 127.6.209.254:44331
I20260926 15:15:45.782874 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 13178
I20260926 15:15:45.784126 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:45.767965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.0.0.1:36150)
0926 15:15:45.769196 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:45.769219 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:45.769420 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:45.769927 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:45.769931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:45.770420 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:45.770670 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:45.770681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.771872 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.771877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:45.773257 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:45.773269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.774335 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.774340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:45.774921 (+ 581us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:45.775572 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:45.775591 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:45.776494 (+ 903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:45.780385 (+ 3891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:45.780392 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:45.780400 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:45.780977 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:45.781466 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:45.781471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:45.781475 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:45.781654 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:45.782161 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:45.782165 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:45.782325 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 15:15:45.782447 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 15:15:45.782708 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1096,"thread_start_us":980,"threads_started":1}
I20260926 15:15:45.784675 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:45.767963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36150 (local address 127.6.209.254:44331)
0926 15:15:45.768145 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:45.768151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:45.769307 (+ 1156us) server_negotiation.cc:407] Connection header received
0926 15:15:45.769585 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:45.769591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:45.769675 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:45.769740 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:45.770815 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.771712 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.773395 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:45.774037 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:45.774119 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:45.776671 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:45.776677 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:45.776682 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:45.776714 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:45.779941 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:45.781124 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:45.781132 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:45.781137 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:45.781317 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:45.781776 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:45.781780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:45.781783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:45.782037 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:45.782195 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:45.782556 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 15:15:45.782730 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 15:15:45.785106 13137 tablet_copy_service.cc:140] P 487b6fe6e7264e4db494c8057789998b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 47996a5d983a4193afbd0f1ee829e0f5 ({username='oryx', principal='oryx/127.6.209.252@KRBTEST.COM'} at 127.0.0.1:36150)
I20260926 15:15:45.785295 13137 tablet_copy_service.cc:161] P 487b6fe6e7264e4db494c8057789998b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 47996a5d983a4193afbd0f1ee829e0f5 at {username='oryx', principal='oryx/127.6.209.252@KRBTEST.COM'} at 127.0.0.1:36150: session id = 47996a5d983a4193afbd0f1ee829e0f5-00000000000000000000000000000000
I20260926 15:15:45.787462 13137 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 15:15:45.788949 13676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 15:15:45.793115 13676 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5: tablet copy: Starting download of 0 data blocks...
I20260926 15:15:45.793345 13676 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5: tablet copy: Starting download of 1 WAL segments...
I20260926 15:15:45.795159 13676 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 15:15:45.798039 13137 tablet_copy_service.cc:342] P 487b6fe6e7264e4db494c8057789998b: Request end of tablet copy session 47996a5d983a4193afbd0f1ee829e0f5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.209.252@KRBTEST.COM'} at 127.0.0.1:36150
I20260926 15:15:45.798153 13137 tablet_copy_service.cc:433] P 487b6fe6e7264e4db494c8057789998b: ending tablet copy session 47996a5d983a4193afbd0f1ee829e0f5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 47996a5d983a4193afbd0f1ee829e0f5
I20260926 15:15:45.805510 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:38531
--local_ip_for_outbound_sockets=127.6.209.193
--tserver_master_addrs=127.6.209.254:44331
--webserver_port=32855
--webserver_interface=127.6.209.193
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.193.keytab
--principal=oryx/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:38737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.209.254:44331,127.6.209.253:38809
--tserver_master_addrs=127.6.209.254:44331,127.6.209.253:38809,127.6.209.252:40519 with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:15:45.809810 13676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:45.813776 13884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:45.814755 13886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:45.821859 13888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:45.822499 13676 server_base.cc:1034] running on GCE node
I20260926 15:15:45.822791 13676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:45.823377 13676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:45.824563 13676 hybrid_clock.cc:648] HybridClock initialized: now 1790435745824524 us; error 31 us; skew 500 ppm
I20260926 15:15:45.824764 13676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:45 dist-test-slave-j6q7 krb5kdc[13072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435745, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:45.826829 13676 init.cc:377] Logged in from keytab as oryx/127.6.209.252@KRBTEST.COM (short username oryx)
I20260926 15:15:45.829598 13676 webserver.cc:460] Webserver started at http://127.6.209.252:38943/ using document root <none> and password file <none>
I20260926 15:15:45.829878 13676 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:45.829993 13676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:45.832085 13676 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 15:15:45.833662 13895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:45.834134 13676 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:15:45.834262 13676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/wal
uuid: "47996a5d983a4193afbd0f1ee829e0f5"
format_stamp: "Formatted at 2026-09-26 15:15:45 on dist-test-slave-j6q7"
server_key: "b0bdda558ddec9b3de57c0a1e143cfc5"
server_key_iv: "37a63f8bd0a81d47b71d1b86aa4c1ed2"
server_key_version: "encryptionkey@0"
I20260926 15:15:45.834429 13676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:45.843207 13676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:45.846172 13676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:45.871934 13676 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.252:40519
I20260926 15:15:45.871974 13946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.252:40519 every 8 connection(s)
I20260926 15:15:45.872709 13676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/master-2/data/info.pb
I20260926 15:15:45.879098 13947 sys_catalog.cc:263] Verifying existing consensus state
I20260926 15:15:45.883937 13947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5: Bootstrap starting.
I20260926 15:15:45.901988 13947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:15:45.902410 13947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5: Bootstrap complete.
I20260926 15:15:45.903126 13947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: NON_VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: true } }
I20260926 15:15:45.903343 13947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47996a5d983a4193afbd0f1ee829e0f5, State: Initialized, Role: LEARNER
I20260926 15:15:45.903571 13947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: NON_VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: true } }
I20260926 15:15:45.905138 13950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: NON_VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: true } } }
I20260926 15:15:45.905422 13947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:15:45.905555 13950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [sys.catalog]: This master's current role is: LEARNER
I20260926 15:15:45.920681 13961 catalog_manager.cc:1259] Loaded cluster ID: aaebb9b0e62d4ad5b164b4119414725a
I20260926 15:15:45.920871 13961 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5: loading cluster ID for follower catalog manager: success
I20260926 15:15:45.923048 13961 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5: acquiring CA information for follower catalog manager: success
I20260926 15:15:45.924695 13961 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 15:15:46.159350 13885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:46.159575 13885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:46.159659 13885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:46.166360 13885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:46.166464 13885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:46.166610 13885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:15:46.175355 13885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:38737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.193.keytab
--principal=oryx/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:38531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=32855
--tserver_master_addrs=127.6.209.254:44331,127.6.209.253:38809,127.6.209.252:40519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:46.176378 13885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:46.177748 13885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:46.189679 13967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:46.189883 13969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:46.190215 13966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:46.190632 13885 server_base.cc:1034] running on GCE node
I20260926 15:15:46.191124 13885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:46.191859 13885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:46.193022 13885 hybrid_clock.cc:648] HybridClock initialized: now 1790435746192977 us; error 40 us; skew 500 ppm
I20260926 15:15:46.193243 13885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:46 dist-test-slave-j6q7 krb5kdc[13072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435746, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:46.195827 13885 init.cc:377] Logged in from keytab as oryx/127.6.209.193@KRBTEST.COM (short username oryx)
I20260926 15:15:46.197059 13885 webserver.cc:460] Webserver started at http://127.6.209.193:32855/ using document root <none> and password file <none>
I20260926 15:15:46.197688 13885 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:46.197819 13885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:46.202816 13885 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 15:15:46.205049 13976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:46.206236 13885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:15:46.206385 13885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "6dc3ad57bb66483bbc89eb29244df045"
format_stamp: "Formatted at 2026-09-26 15:15:43 on dist-test-slave-j6q7"
server_key: "0a732365984a31318527f72e36dfbdd5"
server_key_iv: "a0289cf6391ca25d9bb958ee687b179d"
server_key_version: "encryptionkey@0"
I20260926 15:15:46.207242 13885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:46.218827 13885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:46.222919 13885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:46.223090 13885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:46.223771 13885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:46.224769 13885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:46.224908 13885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:46.225014 13885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:46.225101 13885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:46.254181 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:46.240421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57140 (local address 127.6.209.252:40519)
0926 15:15:46.249161 (+ 8740us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:46.249170 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:46.249186 (+ 16us) server_negotiation.cc:407] Connection header received
0926 15:15:46.249240 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:46.249244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:46.249307 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:46.249437 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:46.249817 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.250893 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.252317 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.253077 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.253232 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:46.253454 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:46.253675 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 15:15:46.253825 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8620,"thread_start_us":8495,"threads_started":1}
I20260926 15:15:46.254185 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:46.240357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:40519 (local address 127.0.0.1:57140)
0926 15:15:46.240714 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:46.240730 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:46.240906 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:46.249463 (+ 8557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:46.249468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:46.249495 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:46.249662 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:46.249670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.251039 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.251044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:46.252176 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:46.252186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.253251 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.253257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:46.253478 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:46.253528 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 15:15:46.253660 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 15:15:46.253836 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":182,"threads_started":1}
I20260926 15:15:46.255187 13922 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 15:15:46.255738 13950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 487b6fe6e7264e4db494c8057789998b. Latest consensus state: current_term: 1 leader_uuid: "487b6fe6e7264e4db494c8057789998b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: NON_VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: true } } }
I20260926 15:15:46.256017 13950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [sys.catalog]: This master's current role is: LEARNER
I20260926 15:15:46.261142 13885 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:38531
I20260926 15:15:46.261219 14090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:38531 every 8 connection(s)
I20260926 15:15:46.262440 13885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:15:46.270220 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 13885
I20260926 15:15:46.270536 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 13311
Sep 26 15:15:46 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435746, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.253@KRBTEST.COM
Sep 26 15:15:46 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435746, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM
Sep 26 15:15:46 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.209.194@KRBTEST.COM
Sep 26 15:15:46 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435746, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.252@KRBTEST.COM
I20260926 15:15:46.299728 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:46.265534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:40627 (local address 127.6.209.254:44331)
0926 15:15:46.265676 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:46.265680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:46.278455 (+ 12775us) server_negotiation.cc:407] Connection header received
0926 15:15:46.279135 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:46.279140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:46.279236 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:46.279310 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:46.282600 (+ 3290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.283641 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.286373 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.286894 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.286975 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:46.292475 (+ 5500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:46.292482 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:46.292486 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:46.292525 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:46.296471 (+ 3946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:46.297272 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:46.297279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:46.297282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:46.297415 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:46.298279 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:46.298284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:46.298288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:46.298639 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:46.298753 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:46.299237 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 15:15:46.299457 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 15:15:46.299784 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:46.265319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:43077 (local address 127.6.209.252:40519)
0926 15:15:46.265468 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:46.265472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:46.276628 (+ 11156us) server_negotiation.cc:407] Connection header received
0926 15:15:46.279714 (+ 3086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:46.279719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:46.279783 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:46.279875 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:46.282792 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.283755 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.286367 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.286868 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.286944 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:46.293528 (+ 6584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:46.293533 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:46.293537 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:46.293573 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:46.296579 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:46.297529 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:46.297534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:46.297538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:46.297669 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:46.298279 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:46.298283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:46.298287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:46.298645 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:46.298757 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:46.299306 (+ 549us) server_negotiation.cc:299] Negotiation successful
0926 15:15:46.299505 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 15:15:46.300876 14100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:46.272038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36016 (local address 127.6.209.194:34425)
0926 15:15:46.279530 (+ 7492us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:46.279535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:46.279556 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:15:46.279601 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:46.279605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:46.279817 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:46.279949 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:46.282633 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.284023 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.285765 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.286488 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.286577 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:46.293594 (+ 7017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:46.293607 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:46.293621 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:46.293677 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:46.297065 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:46.297998 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:46.298007 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:46.298013 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:46.298142 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:46.298572 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:46.298579 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:46.298583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:46.298977 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:46.299233 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:46.300147 (+ 914us) server_negotiation.cc:299] Negotiation successful
0926 15:15:46.300358 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7359,"thread_start_us":7217,"threads_started":1}
I20260926 15:15:46.300915 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:46.265315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:38107 (local address 127.6.209.253:38809)
0926 15:15:46.280935 (+ 15620us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:46.280941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:46.280964 (+ 23us) server_negotiation.cc:407] Connection header received
0926 15:15:46.281017 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:46.281025 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:46.281131 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:46.281235 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:46.282626 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.283641 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.286376 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.286894 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.286987 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:46.291934 (+ 4947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:46.291939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:46.291945 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:46.291976 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:46.295342 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:46.296115 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:46.296120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:46.296124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:46.296244 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:46.296704 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:46.296710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:46.296712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:46.297100 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:46.297243 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:46.300220 (+ 2977us) server_negotiation.cc:299] Negotiation successful
0926 15:15:46.300399 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15447,"thread_start_us":16026,"threads_started":1}
I20260926 15:15:46.300974 14097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:46.265989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.6.209.193:40627)
0926 15:15:46.278328 (+ 12339us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:46.278350 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:46.278872 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:46.279766 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:46.279780 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:46.280957 (+ 1177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:46.282419 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:46.282448 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.283845 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.283850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:46.286238 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:46.286249 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.287311 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.287318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:46.287785 (+ 467us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:46.288707 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:46.288747 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:46.292304 (+ 3557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:46.296590 (+ 4286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:46.296594 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:46.296596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:46.297138 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:46.297548 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:46.297552 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:46.297556 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:46.298135 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:46.298822 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:46.298828 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:46.298975 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 15:15:46.299952 (+ 977us) client_negotiation.cc:240] Negotiation successful
0926 15:15:46.300177 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12144,"mutex_wait_us":1289,"spinlock_wait_cycles":34176,"thread_start_us":10497,"threads_started":1}
I20260926 15:15:46.301188 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:46.266304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:38809 (local address 127.6.209.193:38107)
0926 15:15:46.277191 (+ 10887us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:46.277204 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:46.278872 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:46.281475 (+ 2603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:46.281482 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:46.282227 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:46.282484 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:46.282495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.283845 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.283850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:46.286238 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:46.286255 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.287311 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.287318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:46.287790 (+ 472us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:46.288707 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:46.288746 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:46.291740 (+ 2994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:46.295483 (+ 3743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:46.295490 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:46.295503 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:46.295986 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:46.296369 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:46.296373 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:46.296376 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:46.296587 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:46.297318 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:46.297326 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:46.298145 (+ 819us) client_negotiation.cc:769] Sending connection context
0926 15:15:46.298421 (+ 276us) client_negotiation.cc:240] Negotiation successful
0926 15:15:46.298909 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10768,"mutex_wait_us":441,"spinlock_wait_cycles":10624,"thread_start_us":13795,"threads_started":1}
I20260926 15:15:46.302320 13434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:46.302548 14092 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44331
I20260926 15:15:46.302574 14093 heartbeater.cc:347] Connected to a master server at 127.6.209.253:38809
I20260926 15:15:46.302675 14096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:46.265989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:40519 (local address 127.6.209.193:43077)
0926 15:15:46.276495 (+ 10506us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:46.276538 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:46.279600 (+ 3062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:46.280055 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:46.280060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:46.280959 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:46.282646 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:46.282658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.283938 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.283943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:46.286232 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:46.286245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.287367 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.287378 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:46.287775 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:46.288768 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:46.288793 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:46.293335 (+ 4542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:46.296715 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:46.296737 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:46.296741 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:46.297396 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:46.297859 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:46.297867 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:46.297873 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:46.298147 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:46.298829 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:46.298833 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:46.299036 (+ 203us) client_negotiation.cc:769] Sending connection context
0926 15:15:46.300119 (+ 1083us) client_negotiation.cc:240] Negotiation successful
0926 15:15:46.300356 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10258,"mutex_wait_us":427,"spinlock_wait_cycles":18304,"thread_start_us":10029,"threads_started":1}
I20260926 15:15:46.302909 14092 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:46.302915 14093 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:46.303704 14091 heartbeater.cc:347] Connected to a master server at 127.6.209.252:40519
I20260926 15:15:46.303848 14091 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:46.303927 14093 heartbeater.cc:511] Master 127.6.209.253:38809 requested a full tablet report, sending...
I20260926 15:15:46.303965 14092 heartbeater.cc:511] Master 127.6.209.254:44331 requested a full tablet report, sending...
I20260926 15:15:46.304282 14091 heartbeater.cc:511] Master 127.6.209.252:40519 requested a full tablet report, sending...
I20260926 15:15:46.306146 13912 ts_manager.cc:194] Registered new tserver with Master: 6dc3ad57bb66483bbc89eb29244df045 (127.6.209.193:38531)
I20260926 15:15:46.306141 13721 ts_manager.cc:194] Registered new tserver with Master: 6dc3ad57bb66483bbc89eb29244df045 (127.6.209.193:38531)
I20260926 15:15:46.307193 13117 ts_manager.cc:194] Re-registered known tserver with Master: 6dc3ad57bb66483bbc89eb29244df045 (127.6.209.193:38531)
I20260926 15:15:46.307969 13117 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.209.193@KRBTEST.COM'} at 127.6.209.193:40627
I20260926 15:15:46.382287 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 13311
I20260926 15:15:46.397223 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:34425
--local_ip_for_outbound_sockets=127.6.209.194
--tserver_master_addrs=127.6.209.254:44331
--webserver_port=36145
--webserver_interface=127.6.209.194
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.194.keytab
--principal=oryx/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:38737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.209.254:44331,127.6.209.253:38809
--tserver_master_addrs=127.6.209.254:44331,127.6.209.253:38809,127.6.209.252:40519 with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:46.684912 14102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:46.685211 14102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:46.685300 14102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:46.691243 14102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:46.691315 14102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:46.691399 14102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:15:46.699354 14102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:38737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.194.keytab
--principal=oryx/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:34425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=36145
--tserver_master_addrs=127.6.209.254:44331,127.6.209.253:38809,127.6.209.252:40519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:46.700310 14102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:46.701467 14102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:46.712657 14107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:46.712719 14108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:46.712706 14110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:46.713393 14102 server_base.cc:1034] running on GCE node
I20260926 15:15:46.713821 14102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:46.714654 14102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:46.715839 14102 hybrid_clock.cc:648] HybridClock initialized: now 1790435746715806 us; error 27 us; skew 500 ppm
I20260926 15:15:46.716136 14102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:46 dist-test-slave-j6q7 krb5kdc[13072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435746, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:46.719084 14102 init.cc:377] Logged in from keytab as oryx/127.6.209.194@KRBTEST.COM (short username oryx)
I20260926 15:15:46.720559 14102 webserver.cc:460] Webserver started at http://127.6.209.194:36145/ using document root <none> and password file <none>
I20260926 15:15:46.721231 14102 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:46.721385 14102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:46.725665 14102 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:15:46.728070 14117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:46.729233 14102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:15:46.729396 14102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "2adbe25c005c4e93927b3d15df65f065"
format_stamp: "Formatted at 2026-09-26 15:15:43 on dist-test-slave-j6q7"
server_key: "f8a934ca2d19e76c159ef27ba1064ef8"
server_key_iv: "16b9e3f799f4bebc7c2c462b99dca788"
server_key_version: "encryptionkey@0"
I20260926 15:15:46.730186 14102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:46.738363 13874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "487b6fe6e7264e4db494c8057789998b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: NON_VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: true } } }
I20260926 15:15:46.738634 13874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:46.740182 14122 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b: attempting to promote NON_VOTER 47996a5d983a4193afbd0f1ee829e0f5 to VOTER
I20260926 15:15:46.740677 14122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: false } }
I20260926 15:15:46.742646 13922 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [term 1 LEARNER]: Refusing update from remote peer 487b6fe6e7264e4db494c8057789998b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 15:15:46.742733 13731 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [term 1 FOLLOWER]: Refusing update from remote peer 487b6fe6e7264e4db494c8057789998b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 15:15:46.743217 14122 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [LEADER]: Connected to new peer: Peer: permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 15:15:46.743592 14123 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 15:15:46.749085 14122 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 47996a5d983a4193afbd0f1ee829e0f5 (127.6.209.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: false } } }
I20260926 15:15:46.749683 13922 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 47996a5d983a4193afbd0f1ee829e0f5 (127.6.209.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: false } } }
I20260926 15:15:46.752053 14123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "487b6fe6e7264e4db494c8057789998b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: false } } }
I20260926 15:15:46.752348 14123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:46.752596 14123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "487b6fe6e7264e4db494c8057789998b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: false } } }
I20260926 15:15:46.752647 14122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "487b6fe6e7264e4db494c8057789998b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: false } } }
I20260926 15:15:46.752859 14123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:46.752887 14122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:46.753191 13731 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 47996a5d983a4193afbd0f1ee829e0f5 (127.6.209.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: false } } }
I20260926 15:15:46.753675 13950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "487b6fe6e7264e4db494c8057789998b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: false } } }
I20260926 15:15:46.753924 13950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:15:46.756057 14128 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "487b6fe6e7264e4db494c8057789998b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: false } } }
I20260926 15:15:46.756376 14128 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:15:46.757102 14102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:46.760779 14102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:46.761094 14102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:46.761910 14102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:46.762990 14102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:46.763065 14102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:46.763137 14102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:46.763185 14102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:46.793567 14102 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:34425
I20260926 15:15:46.793592 14236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:34425 every 8 connection(s)
I20260926 15:15:46.794761 14102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:15:46.803107 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 14102
I20260926 15:15:46.803385 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 13444
Sep 26 15:15:46 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435746, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.253@KRBTEST.COM
Sep 26 15:15:46 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435746, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM
Sep 26 15:15:46 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435746, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.252@KRBTEST.COM
I20260926 15:15:46.820628 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:46.798171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:58027 (local address 127.6.209.254:44331)
0926 15:15:46.798369 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:46.798377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:46.800798 (+ 2421us) server_negotiation.cc:407] Connection header received
0926 15:15:46.802343 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:46.802349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:46.802424 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:46.802529 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:46.804910 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.805849 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.808425 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.809073 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.809187 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:46.813489 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:46.813494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:46.813499 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:46.813531 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:46.816630 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:46.818392 (+ 1762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:46.818397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:46.818400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:46.818540 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:46.818998 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:46.819004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:46.819008 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:46.819288 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:46.819419 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:46.820103 (+ 684us) server_negotiation.cc:299] Negotiation successful
0926 15:15:46.820322 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 15:15:46.821034 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:46.798794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:35371 (local address 127.6.209.252:40519)
0926 15:15:46.798998 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:46.799005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:46.799486 (+ 481us) server_negotiation.cc:407] Connection header received
0926 15:15:46.801911 (+ 2425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:46.801918 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:46.802006 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:46.802079 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:46.805062 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.805972 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.808102 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.808690 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.808771 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:46.814112 (+ 5341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:46.814118 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:46.814122 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:46.814157 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:46.817234 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:46.818353 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:46.818359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:46.818362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:46.818509 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:46.819034 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:46.819039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:46.819042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:46.819733 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:46.819864 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:46.820395 (+ 531us) server_negotiation.cc:299] Negotiation successful
0926 15:15:46.820633 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Sep 26 15:15:46 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.209.195@KRBTEST.COM
I20260926 15:15:46.822309 14242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:46.798408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.6.209.194:58027)
0926 15:15:46.800690 (+ 2282us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:46.800719 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:46.801622 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:46.802567 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:46.802573 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:46.803831 (+ 1258us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:46.804464 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:46.804470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.806009 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.806016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:46.808302 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:46.808314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.809251 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.809258 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:46.809422 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:46.809997 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:46.810019 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:46.813282 (+ 3263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:46.817416 (+ 4134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:46.817421 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:46.817425 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:46.818235 (+ 810us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:46.818630 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:46.818635 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:46.818639 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:46.818904 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:46.819393 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:46.819400 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:46.819780 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 15:15:46.820029 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 15:15:46.820446 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2195,"mutex_wait_us":574,"spinlock_wait_cycles":2816,"thread_start_us":592,"threads_started":1}
I20260926 15:15:46.822664 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:46.797316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:45435 (local address 127.6.209.253:38809)
0926 15:15:46.797497 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:46.797504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:46.800330 (+ 2826us) server_negotiation.cc:407] Connection header received
0926 15:15:46.801963 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:46.801968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:46.802087 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:46.802211 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:46.804398 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.805264 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.808072 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.808634 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.808741 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:46.813111 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:46.813117 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:46.813121 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:46.813153 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:46.818236 (+ 5083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:46.819734 (+ 1498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:46.819738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:46.819741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:46.819862 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:46.821172 (+ 1310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:46.821179 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:46.821185 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:46.821531 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:46.821645 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:46.822170 (+ 525us) server_negotiation.cc:299] Negotiation successful
0926 15:15:46.822372 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 15:15:46.822356 14243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:46.797918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:40519 (local address 127.6.209.194:35371)
0926 15:15:46.799310 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:46.799358 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:46.801622 (+ 2264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:46.802379 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:46.802386 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:46.803664 (+ 1278us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:46.804936 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:46.804946 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.806144 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.806151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:46.807948 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:46.807963 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.809045 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.809051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:46.809205 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:46.809804 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:46.809828 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:46.813941 (+ 4113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:46.817376 (+ 3435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:46.817385 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:46.817400 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:46.818188 (+ 788us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:46.818643 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:46.818647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:46.818651 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:46.818912 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:46.819941 (+ 1029us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:46.819946 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:46.820091 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 15:15:46.821267 (+ 1176us) client_negotiation.cc:240] Negotiation successful
0926 15:15:46.821531 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"mutex_wait_us":953,"spinlock_wait_cycles":3840,"thread_start_us":1944,"threads_started":1}
I20260926 15:15:46.823918 14237 heartbeater.cc:347] Connected to a master server at 127.6.209.252:40519
I20260926 15:15:46.823973 14238 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44331
I20260926 15:15:46.824183 14237 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:46.824188 14238 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:46.825001 14237 heartbeater.cc:511] Master 127.6.209.252:40519 requested a full tablet report, sending...
I20260926 15:15:46.825026 14238 heartbeater.cc:511] Master 127.6.209.254:44331 requested a full tablet report, sending...
I20260926 15:15:46.826524 13117 ts_manager.cc:194] Re-registered known tserver with Master: 2adbe25c005c4e93927b3d15df65f065 (127.6.209.194:34425)
I20260926 15:15:46.827420 13912 ts_manager.cc:194] Registered new tserver with Master: 2adbe25c005c4e93927b3d15df65f065 (127.6.209.194:34425)
I20260926 15:15:46.827498 13117 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.209.194@KRBTEST.COM'} at 127.6.209.194:58027
I20260926 15:15:46.828009 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:46.797918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:38809 (local address 127.6.209.194:45435)
0926 15:15:46.800233 (+ 2315us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:46.800259 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:46.801624 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:46.802382 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:46.802387 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:46.803501 (+ 1114us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:46.804279 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:46.804295 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.805396 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.805404 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:46.807940 (+ 2536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:46.807951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.808852 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.808859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:46.809062 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:46.809686 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:46.809711 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:46.812885 (+ 3174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:46.818678 (+ 5793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:46.818683 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:46.818687 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:46.819622 (+ 935us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:46.819988 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:46.820859 (+ 871us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:46.820865 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:46.821033 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:46.821741 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:46.821746 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:46.821885 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 15:15:46.822861 (+ 976us) client_negotiation.cc:240] Negotiation successful
0926 15:15:46.823327 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2177,"mutex_wait_us":118,"spinlock_wait_cycles":58368,"thread_start_us":2420,"threads_started":1}
I20260926 15:15:46.829519 14245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:46.804370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37758 (local address 127.6.209.195:44185)
0926 15:15:46.804911 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:46.804918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:46.808194 (+ 3276us) server_negotiation.cc:407] Connection header received
0926 15:15:46.808257 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:46.808264 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:46.810090 (+ 1826us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:46.810234 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:46.812722 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.814931 (+ 2209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.816229 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:46.816796 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:46.816938 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:46.822694 (+ 5756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:46.822708 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:46.822722 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:46.822772 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:46.826335 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:46.827251 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:46.827259 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:46.827263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:46.827390 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:46.827849 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:46.827855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:46.827858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:46.828246 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:46.828568 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:46.828998 (+ 430us) server_negotiation.cc:299] Negotiation successful
0926 15:15:46.829166 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":245,"threads_started":1}
I20260926 15:15:46.830178 14239 heartbeater.cc:347] Connected to a master server at 127.6.209.253:38809
I20260926 15:15:46.830296 14239 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:46.830559 14239 heartbeater.cc:511] Master 127.6.209.253:38809 requested a full tablet report, sending...
I20260926 15:15:46.830580 13567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:46.831470 13721 ts_manager.cc:194] Registered new tserver with Master: 2adbe25c005c4e93927b3d15df65f065 (127.6.209.194:34425)
I20260926 15:15:46.906865 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 13444
I20260926 15:15:46.922071 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:44185
--local_ip_for_outbound_sockets=127.6.209.195
--tserver_master_addrs=127.6.209.254:44331
--webserver_port=37689
--webserver_interface=127.6.209.195
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.195.keytab
--principal=oryx/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:38737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.209.254:44331,127.6.209.253:38809
--tserver_master_addrs=127.6.209.254:44331,127.6.209.253:38809,127.6.209.252:40519 with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:47.214409 14247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:47.214610 14247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:47.214754 14247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:47.220978 14247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:47.221082 14247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:47.221236 14247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:15:47.229972 14247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:38737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.195.keytab
--principal=oryx/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:44185
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=37689
--tserver_master_addrs=127.6.209.254:44331,127.6.209.253:38809,127.6.209.252:40519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:47.231031 14247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:47.232146 14247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:47.244067 14255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:47.244155 14253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:47.244096 14252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:47.247551 14247 server_base.cc:1034] running on GCE node
I20260926 15:15:47.248057 14247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:47.248903 14247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:47.250099 14247 hybrid_clock.cc:648] HybridClock initialized: now 1790435747250065 us; error 41 us; skew 500 ppm
I20260926 15:15:47.250361 14247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:47 dist-test-slave-j6q7 krb5kdc[13072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435747, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:47.253159 14247 init.cc:377] Logged in from keytab as oryx/127.6.209.195@KRBTEST.COM (short username oryx)
I20260926 15:15:47.254365 14247 webserver.cc:460] Webserver started at http://127.6.209.195:37689/ using document root <none> and password file <none>
I20260926 15:15:47.255043 14247 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:47.255141 14247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:47.259624 14247 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:15:47.262013 14262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:47.263139 14247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:15:47.263278 14247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "ac66294fc8734840a60cbe11e4625260"
format_stamp: "Formatted at 2026-09-26 15:15:44 on dist-test-slave-j6q7"
server_key: "5cd3210ffe46dddb6ff3f19f7056510d"
server_key_iv: "ecf0ded2cd40b7b71c0598703c263374"
server_key_version: "encryptionkey@0"
I20260926 15:15:47.264056 14247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:47.279124 14247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:47.282667 14247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:47.282840 14247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:47.283598 14247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:47.284705 14247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:47.284806 14247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:47.284937 14247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:47.285032 14247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:47.310685 14092 heartbeater.cc:503] Master 127.6.209.254:44331 was elected leader, sending a full tablet report...
I20260926 15:15:47.314894 14247 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:44185
I20260926 15:15:47.314957 14374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:44185 every 8 connection(s)
I20260926 15:15:47.316045 14247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:15:47.318051 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 14247
Sep 26 15:15:47 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435747, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.252@KRBTEST.COM
Sep 26 15:15:47 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435747, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM
Sep 26 15:15:47 dist-test-slave-j6q7 krb5kdc[13072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435747, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.253@KRBTEST.COM
I20260926 15:15:47.354895 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:47.319818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:51569 (local address 127.6.209.252:40519)
0926 15:15:47.320190 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:47.320197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:47.329193 (+ 8996us) server_negotiation.cc:407] Connection header received
0926 15:15:47.337469 (+ 8276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:47.337474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:47.337551 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:47.337644 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:47.339486 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.340524 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.342485 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.342890 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.343044 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:47.346846 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:47.346869 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:47.346874 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:47.346910 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:47.350769 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:47.352536 (+ 1767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:47.352542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:47.352545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:47.352663 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:47.353163 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:47.353168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:47.353171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:47.353464 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:47.353571 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:47.354393 (+ 822us) server_negotiation.cc:299] Negotiation successful
0926 15:15:47.354593 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256}
I20260926 15:15:47.356825 14381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:47.319298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.6.209.195:46699)
0926 15:15:47.331735 (+ 12437us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:47.331764 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:47.336171 (+ 4407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:47.337073 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:47.337087 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:47.338270 (+ 1183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:47.339298 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:47.339316 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.341146 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.341155 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:47.343559 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:47.343578 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.344973 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.344981 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:47.345141 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:47.346862 (+ 1721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:47.346882 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:47.349718 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:47.353547 (+ 3829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:47.353551 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:47.353554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:47.354001 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:47.354387 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:47.354391 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:47.354394 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:47.354533 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:47.355101 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:47.355105 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:47.355219 (+ 114us) client_negotiation.cc:769] Sending connection context
0926 15:15:47.355324 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 15:15:47.355568 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12285,"mutex_wait_us":364,"spinlock_wait_cycles":25984,"thread_start_us":13306,"threads_started":1}
I20260926 15:15:47.357120 14384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:47.319299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:40519 (local address 127.6.209.195:51569)
0926 15:15:47.329025 (+ 9726us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:47.329078 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:47.336171 (+ 7093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:47.337807 (+ 1636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:47.337812 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:47.338342 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:47.339275 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:47.339304 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.340697 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.340702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:47.342353 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:47.342364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.343127 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.343136 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:47.343386 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:47.344023 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:47.344043 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:47.346627 (+ 2584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:47.350888 (+ 4261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:47.350893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:47.350902 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:47.352407 (+ 1505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:47.352789 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:47.352793 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:47.352800 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:47.353048 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:47.353647 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:47.353654 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:47.354001 (+ 347us) client_negotiation.cc:769] Sending connection context
0926 15:15:47.355101 (+ 1100us) client_negotiation.cc:240] Negotiation successful
0926 15:15:47.355649 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9374,"mutex_wait_us":172,"thread_start_us":12048,"threads_started":1}
I20260926 15:15:47.357990 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:47.319660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:46699 (local address 127.6.209.254:44331)
0926 15:15:47.320049 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:47.320057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:47.331846 (+ 11789us) server_negotiation.cc:407] Connection header received
0926 15:15:47.336457 (+ 4611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:47.336465 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:47.336559 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:47.336628 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:47.339542 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.340929 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.343765 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.344638 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.344759 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:47.349875 (+ 5116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:47.349881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:47.349886 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:47.349919 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:47.353426 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:47.354136 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:47.354140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:47.354143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:47.354281 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:47.354695 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:47.354700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:47.354703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:47.354983 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:47.357267 (+ 2284us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:47.357414 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 15:15:47.357628 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260926 15:15:47.359063 14375 heartbeater.cc:347] Connected to a master server at 127.6.209.252:40519
I20260926 15:15:47.359398 14375 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:47.360314 14375 heartbeater.cc:511] Master 127.6.209.252:40519 requested a full tablet report, sending...
I20260926 15:15:47.362375 13912 ts_manager.cc:194] Registered new tserver with Master: ac66294fc8734840a60cbe11e4625260 (127.6.209.195:44185)
I20260926 15:15:47.363777 14376 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44331
I20260926 15:15:47.363884 14376 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:47.364244 14376 heartbeater.cc:511] Master 127.6.209.254:44331 requested a full tablet report, sending...
I20260926 15:15:47.365039 13117 ts_manager.cc:194] Re-registered known tserver with Master: ac66294fc8734840a60cbe11e4625260 (127.6.209.195:44185)
I20260926 15:15:47.366117 13117 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.209.195@KRBTEST.COM'} at 127.6.209.195:46699
I20260926 15:15:47.368356 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:47.325287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:38809 (local address 127.6.209.195:48787)
0926 15:15:47.333200 (+ 7913us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:47.333218 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:47.336183 (+ 2965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:47.339129 (+ 2946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:47.339137 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:47.339703 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:47.339935 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:47.339944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.341371 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.341375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:47.346616 (+ 5241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:47.346625 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.348004 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.348018 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:47.349015 (+ 997us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:47.349525 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:47.349544 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:47.354629 (+ 5085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:47.365347 (+ 10718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:47.365351 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:47.365354 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:47.365817 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:47.366204 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:47.366211 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:47.366216 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:47.366397 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:47.367330 (+ 933us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:47.367334 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:47.367457 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 15:15:47.367599 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 15:15:47.367927 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7811,"mutex_wait_us":4,"thread_start_us":3243,"threads_started":1}
I20260926 15:15:47.369364 14383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:47.325098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:48787 (local address 127.6.209.253:38809)
0926 15:15:47.337134 (+ 12036us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:47.337143 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:47.337159 (+ 16us) server_negotiation.cc:407] Connection header received
0926 15:15:47.338882 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:47.338887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:47.338956 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:47.340056 (+ 1100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:47.340173 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.341223 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.346764 (+ 5541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.347741 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.349261 (+ 1520us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:47.354848 (+ 5587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:47.354856 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:47.354863 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:47.354904 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:47.365219 (+ 10315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:47.365952 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:47.365957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:47.365960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:47.366083 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:47.366836 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:47.366840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:47.366843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:47.367161 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:47.368682 (+ 1521us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:47.368836 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 15:15:47.369017 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11920,"thread_start_us":11838,"threads_started":1}
I20260926 15:15:47.370288 14377 heartbeater.cc:347] Connected to a master server at 127.6.209.253:38809
I20260926 15:15:47.370395 14377 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:47.370690 14377 heartbeater.cc:511] Master 127.6.209.253:38809 requested a full tablet report, sending...
I20260926 15:15:47.371589 13721 ts_manager.cc:194] Registered new tserver with Master: ac66294fc8734840a60cbe11e4625260 (127.6.209.195:44185)
I20260926 15:15:47.378288 14390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:47.356827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34810 (local address 127.6.209.253:38809)
0926 15:15:47.357611 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:47.357617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:47.358871 (+ 1254us) server_negotiation.cc:407] Connection header received
0926 15:15:47.359102 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:47.359107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:47.359164 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:47.359232 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:47.361926 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.363193 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.368742 (+ 5549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.369345 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.369429 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:47.372679 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:47.372685 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:47.372688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:47.372720 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:47.375702 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:47.376345 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:47.376350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:47.376353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:47.376471 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:47.376871 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:47.376876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:47.376880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:47.377189 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:47.377457 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:47.377744 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 15:15:47.377946 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":521,"threads_started":1}
I20260926 15:15:47.381058 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:47.359449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57144 (local address 127.6.209.252:40519)
0926 15:15:47.359613 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:47.359619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:47.365262 (+ 5643us) server_negotiation.cc:407] Connection header received
0926 15:15:47.365996 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:47.366002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:47.366071 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:47.366139 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:47.368688 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.369618 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.371450 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.371990 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.372075 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:47.374739 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:47.374744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:47.374749 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:47.374781 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:47.378549 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:47.379150 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:47.379156 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:47.379159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:47.379288 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:47.379682 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:47.379689 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:47.379693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:47.380005 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:47.380147 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:47.380473 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 15:15:47.380646 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 15:15:47.381306 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:47.350327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36156 (local address 127.6.209.254:44331)
0926 15:15:47.358678 (+ 8351us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:47.358682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:47.358697 (+ 15us) server_negotiation.cc:407] Connection header received
0926 15:15:47.358735 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:47.358738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:47.362789 (+ 4051us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:47.362869 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:47.367789 (+ 4920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.368936 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.370819 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.371483 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.371578 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:47.374824 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:47.374832 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:47.374837 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:47.374871 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:47.378437 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:47.379476 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:47.379484 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:47.379489 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:47.379611 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:47.379918 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:47.379920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:47.379922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:47.380183 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:47.380377 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:47.380698 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 15:15:47.380958 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8273,"thread_start_us":12882,"threads_started":1}
I20260926 15:15:47.384249 13117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:15:47.390901 13117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:15:47.428776 14402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:47.404696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36022 (local address 127.6.209.194:34425)
0926 15:15:47.422311 (+ 17615us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:47.422319 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:47.422348 (+ 29us) server_negotiation.cc:407] Connection header received
0926 15:15:47.422443 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:47.422448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:47.422636 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:47.422818 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:47.423356 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.424483 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.426461 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.427735 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.427878 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:47.428074 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:47.428328 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 15:15:47.428527 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17446,"thread_start_us":17350,"threads_started":1}
I20260926 15:15:47.429080 14405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:47.404491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:34425 (local address 127.0.0.1:36022)
0926 15:15:47.414975 (+ 10484us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:47.414998 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:47.415137 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:47.422804 (+ 7667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:47.422809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:47.422833 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:47.423169 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:47.423186 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.424643 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.424647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:47.426282 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:47.426293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.427939 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.427945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:47.428140 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:47.428177 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 15:15:47.428287 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 15:15:47.428525 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10338,"mutex_wait_us":95,"thread_start_us":19232,"threads_started":1}
I20260926 15:15:47.432013 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:47.404928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:44185 (local address 127.0.0.1:37772)
0926 15:15:47.423385 (+ 18457us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:47.423397 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:47.423512 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:47.424557 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:47.424562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:47.424590 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:47.424952 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:47.424962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.426730 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.426734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:47.429664 (+ 2930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:47.429677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.430916 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.430922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:47.431380 (+ 458us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:47.431458 (+ 78us) client_negotiation.cc:769] Sending connection context
0926 15:15:47.431530 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 15:15:47.431641 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18337,"mutex_wait_us":47,"thread_start_us":18204,"threads_started":1}
I20260926 15:15:47.432386 14401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:47.405357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:38531 (local address 127.0.0.1:46810)
0926 15:15:47.423956 (+ 18599us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:47.423972 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:47.424492 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:47.425354 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:47.425361 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:47.425386 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:47.425871 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:47.425883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.427883 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.427890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:47.429929 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:47.429941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.431622 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.431627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:47.431776 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:47.431802 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 15:15:47.431883 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 15:15:47.431980 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18499,"mutex_wait_us":23,"thread_start_us":9872,"threads_started":1}
I20260926 15:15:47.432859 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:47.405935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46810 (local address 127.6.209.193:38531)
0926 15:15:47.421317 (+ 15382us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:47.421329 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:47.424474 (+ 3145us) server_negotiation.cc:407] Connection header received
0926 15:15:47.424691 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:47.424699 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:47.424930 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:47.425074 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:47.426055 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.427497 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.430077 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.431218 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.431323 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:47.431476 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:47.432245 (+ 769us) server_negotiation.cc:299] Negotiation successful
0926 15:15:47.432389 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15202,"thread_start_us":15088,"threads_started":1}
I20260926 15:15:47.433041 14404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:47.405267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37772 (local address 127.6.209.195:44185)
0926 15:15:47.421882 (+ 16615us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:47.421895 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:47.423479 (+ 1584us) server_negotiation.cc:407] Connection header received
0926 15:15:47.423664 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:47.423674 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:47.423904 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:47.424504 (+ 600us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:47.425119 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.426573 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.429831 (+ 3258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.430731 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.430868 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:47.431057 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:47.432260 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0926 15:15:47.432686 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16393,"thread_start_us":16214,"threads_started":1}
I20260926 15:15:47.432452 14172 tablet_service.cc:1467] Processing CreateTablet for tablet e3809eab701e47138431af019106ab8d (DEFAULT_TABLE table=test-table [id=330397a910494126b441d5d4deafffef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:47.433871 14172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3809eab701e47138431af019106ab8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:47.436322 14310 tablet_service.cc:1467] Processing CreateTablet for tablet e3809eab701e47138431af019106ab8d (DEFAULT_TABLE table=test-table [id=330397a910494126b441d5d4deafffef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:47.439069 14024 tablet_service.cc:1467] Processing CreateTablet for tablet e3809eab701e47138431af019106ab8d (DEFAULT_TABLE table=test-table [id=330397a910494126b441d5d4deafffef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:47.440526 14024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3809eab701e47138431af019106ab8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:47.441296 14310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3809eab701e47138431af019106ab8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:47.448055 14407 tablet_bootstrap.cc:492] T e3809eab701e47138431af019106ab8d P 2adbe25c005c4e93927b3d15df65f065: Bootstrap starting.
I20260926 15:15:47.451359 14407 tablet_bootstrap.cc:654] T e3809eab701e47138431af019106ab8d P 2adbe25c005c4e93927b3d15df65f065: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:47.452381 14408 tablet_bootstrap.cc:492] T e3809eab701e47138431af019106ab8d P 6dc3ad57bb66483bbc89eb29244df045: Bootstrap starting.
I20260926 15:15:47.452945 14407 log.cc:824] T e3809eab701e47138431af019106ab8d P 2adbe25c005c4e93927b3d15df65f065: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:47.453866 14409 tablet_bootstrap.cc:492] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260: Bootstrap starting.
I20260926 15:15:47.455209 14408 tablet_bootstrap.cc:654] T e3809eab701e47138431af019106ab8d P 6dc3ad57bb66483bbc89eb29244df045: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:47.456300 14408 log.cc:824] T e3809eab701e47138431af019106ab8d P 6dc3ad57bb66483bbc89eb29244df045: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:47.457157 14409 tablet_bootstrap.cc:654] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:47.457479 14407 tablet_bootstrap.cc:492] T e3809eab701e47138431af019106ab8d P 2adbe25c005c4e93927b3d15df65f065: No bootstrap required, opened a new log
I20260926 15:15:47.457767 14407 ts_tablet_manager.cc:1397] T e3809eab701e47138431af019106ab8d P 2adbe25c005c4e93927b3d15df65f065: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 15:15:47.458268 14409 log.cc:824] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:47.459023 14408 tablet_bootstrap.cc:492] T e3809eab701e47138431af019106ab8d P 6dc3ad57bb66483bbc89eb29244df045: No bootstrap required, opened a new log
I20260926 15:15:47.459242 14408 ts_tablet_manager.cc:1397] T e3809eab701e47138431af019106ab8d P 6dc3ad57bb66483bbc89eb29244df045: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 15:15:47.460709 14409 tablet_bootstrap.cc:492] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260: No bootstrap required, opened a new log
I20260926 15:15:47.461012 14409 ts_tablet_manager.cc:1397] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260926 15:15:47.464967 14407 raft_consensus.cc:348] T e3809eab701e47138431af019106ab8d P 2adbe25c005c4e93927b3d15df65f065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adbe25c005c4e93927b3d15df65f065" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34425 } } peers { permanent_uuid: "ac66294fc8734840a60cbe11e4625260" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44185 } } peers { permanent_uuid: "6dc3ad57bb66483bbc89eb29244df045" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38531 } }
I20260926 15:15:47.465269 14407 raft_consensus.cc:374] T e3809eab701e47138431af019106ab8d P 2adbe25c005c4e93927b3d15df65f065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:47.465379 14407 raft_consensus.cc:729] T e3809eab701e47138431af019106ab8d P 2adbe25c005c4e93927b3d15df65f065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2adbe25c005c4e93927b3d15df65f065, State: Initialized, Role: FOLLOWER
I20260926 15:15:47.465898 14407 consensus_queue.cc:260] T e3809eab701e47138431af019106ab8d P 2adbe25c005c4e93927b3d15df65f065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adbe25c005c4e93927b3d15df65f065" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34425 } } peers { permanent_uuid: "ac66294fc8734840a60cbe11e4625260" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44185 } } peers { permanent_uuid: "6dc3ad57bb66483bbc89eb29244df045" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38531 } }
I20260926 15:15:47.467274 14238 heartbeater.cc:503] Master 127.6.209.254:44331 was elected leader, sending a full tablet report...
I20260926 15:15:47.470165 14409 raft_consensus.cc:348] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adbe25c005c4e93927b3d15df65f065" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34425 } } peers { permanent_uuid: "ac66294fc8734840a60cbe11e4625260" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44185 } } peers { permanent_uuid: "6dc3ad57bb66483bbc89eb29244df045" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38531 } }
I20260926 15:15:47.470512 14409 raft_consensus.cc:374] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:47.470582 14409 raft_consensus.cc:729] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac66294fc8734840a60cbe11e4625260, State: Initialized, Role: FOLLOWER
I20260926 15:15:47.471050 14409 consensus_queue.cc:260] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adbe25c005c4e93927b3d15df65f065" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34425 } } peers { permanent_uuid: "ac66294fc8734840a60cbe11e4625260" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44185 } } peers { permanent_uuid: "6dc3ad57bb66483bbc89eb29244df045" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38531 } }
I20260926 15:15:47.471206 14408 raft_consensus.cc:348] T e3809eab701e47138431af019106ab8d P 6dc3ad57bb66483bbc89eb29244df045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adbe25c005c4e93927b3d15df65f065" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34425 } } peers { permanent_uuid: "ac66294fc8734840a60cbe11e4625260" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44185 } } peers { permanent_uuid: "6dc3ad57bb66483bbc89eb29244df045" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38531 } }
I20260926 15:15:47.471560 14408 raft_consensus.cc:374] T e3809eab701e47138431af019106ab8d P 6dc3ad57bb66483bbc89eb29244df045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:47.471660 14408 raft_consensus.cc:729] T e3809eab701e47138431af019106ab8d P 6dc3ad57bb66483bbc89eb29244df045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc3ad57bb66483bbc89eb29244df045, State: Initialized, Role: FOLLOWER
I20260926 15:15:47.472255 14407 ts_tablet_manager.cc:1428] T e3809eab701e47138431af019106ab8d P 2adbe25c005c4e93927b3d15df65f065: Time spent starting tablet: real 0.014s user 0.010s sys 0.000s
I20260926 15:15:47.472275 14408 consensus_queue.cc:260] T e3809eab701e47138431af019106ab8d P 6dc3ad57bb66483bbc89eb29244df045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adbe25c005c4e93927b3d15df65f065" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34425 } } peers { permanent_uuid: "ac66294fc8734840a60cbe11e4625260" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44185 } } peers { permanent_uuid: "6dc3ad57bb66483bbc89eb29244df045" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38531 } }
I20260926 15:15:47.473320 14376 heartbeater.cc:503] Master 127.6.209.254:44331 was elected leader, sending a full tablet report...
I20260926 15:15:47.473869 14409 ts_tablet_manager.cc:1428] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260926 15:15:47.481128 14408 ts_tablet_manager.cc:1428] T e3809eab701e47138431af019106ab8d P 6dc3ad57bb66483bbc89eb29244df045: Time spent starting tablet: real 0.022s user 0.014s sys 0.005s
W20260926 15:15:47.520982 14094 tablet.cc:2369] T e3809eab701e47138431af019106ab8d P 6dc3ad57bb66483bbc89eb29244df045: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:15:47.548542 14240 tablet.cc:2369] T e3809eab701e47138431af019106ab8d P 2adbe25c005c4e93927b3d15df65f065: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:15:47.584309 14378 tablet.cc:2369] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:15:47.603655 14414 raft_consensus.cc:482] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:15:47.604259 14414 raft_consensus.cc:504] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adbe25c005c4e93927b3d15df65f065" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34425 } } peers { permanent_uuid: "ac66294fc8734840a60cbe11e4625260" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44185 } } peers { permanent_uuid: "6dc3ad57bb66483bbc89eb29244df045" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38531 } }
I20260926 15:15:47.605901 14414 leader_election.cc:288] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2adbe25c005c4e93927b3d15df65f065 (127.6.209.194:34425), 6dc3ad57bb66483bbc89eb29244df045 (127.6.209.193:38531)
I20260926 15:15:47.611115 14402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:47.606459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:51987 (local address 127.6.209.194:34425)
0926 15:15:47.606607 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:47.606610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:47.606623 (+ 13us) server_negotiation.cc:407] Connection header received
0926 15:15:47.606720 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:47.606723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:47.606775 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:47.606866 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:47.607195 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.607885 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.609498 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.610170 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.610354 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:47.610422 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:47.610766 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 15:15:47.610879 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 15:15:47.611157 14384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:47.606118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:34425 (local address 127.6.209.195:51987)
0926 15:15:47.606355 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:47.606373 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:47.606481 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:47.606879 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:47.606883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:47.606911 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:47.607066 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:47.607073 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.608004 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.608013 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:47.609365 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:47.609375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.610343 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.610348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:47.610572 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:47.610620 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 15:15:47.610733 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 15:15:47.610862 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":6,"spinlock_wait_cycles":4864}
I20260926 15:15:47.611188 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:47.606285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:42121 (local address 127.6.209.193:38531)
0926 15:15:47.606454 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:47.606459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:47.606477 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:15:47.606644 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:47.606648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:47.606699 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:47.606792 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:47.607145 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.607858 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.609490 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.610243 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.610413 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:47.610480 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:47.610840 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 15:15:47.610952 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:15:47.611261 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:47.606148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:38531 (local address 127.6.209.195:42121)
0926 15:15:47.606357 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:47.606373 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:47.606481 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:47.606818 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:47.606822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:47.606845 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:47.607017 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:47.607024 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.607987 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.607993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:47.609352 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:47.609363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.610452 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.610457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:47.610699 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:47.610724 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 15:15:47.610818 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 15:15:47.610947 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":54,"spinlock_wait_cycles":6656}
I20260926 15:15:47.612105 14046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3809eab701e47138431af019106ab8d" candidate_uuid: "ac66294fc8734840a60cbe11e4625260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc3ad57bb66483bbc89eb29244df045" is_pre_election: true
I20260926 15:15:47.612130 14192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3809eab701e47138431af019106ab8d" candidate_uuid: "ac66294fc8734840a60cbe11e4625260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2adbe25c005c4e93927b3d15df65f065" is_pre_election: true
I20260926 15:15:47.612548 14046 raft_consensus.cc:2454] T e3809eab701e47138431af019106ab8d P 6dc3ad57bb66483bbc89eb29244df045 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac66294fc8734840a60cbe11e4625260 in term 0.
I20260926 15:15:47.612564 14192 raft_consensus.cc:2454] T e3809eab701e47138431af019106ab8d P 2adbe25c005c4e93927b3d15df65f065 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac66294fc8734840a60cbe11e4625260 in term 0.
I20260926 15:15:47.613188 14263 leader_election.cc:302] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2adbe25c005c4e93927b3d15df65f065, 6dc3ad57bb66483bbc89eb29244df045, ac66294fc8734840a60cbe11e4625260; no voters:
I20260926 15:15:47.613540 14414 raft_consensus.cc:2781] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:15:47.613672 14414 raft_consensus.cc:482] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:15:47.613792 14414 raft_consensus.cc:3037] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:47.615688 14414 raft_consensus.cc:504] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adbe25c005c4e93927b3d15df65f065" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34425 } } peers { permanent_uuid: "ac66294fc8734840a60cbe11e4625260" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44185 } } peers { permanent_uuid: "6dc3ad57bb66483bbc89eb29244df045" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38531 } }
I20260926 15:15:47.616159 14414 leader_election.cc:288] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260 [CANDIDATE]: Term 1 election: Requested vote from peers 2adbe25c005c4e93927b3d15df65f065 (127.6.209.194:34425), 6dc3ad57bb66483bbc89eb29244df045 (127.6.209.193:38531)
I20260926 15:15:47.616537 14046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3809eab701e47138431af019106ab8d" candidate_uuid: "ac66294fc8734840a60cbe11e4625260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc3ad57bb66483bbc89eb29244df045"
I20260926 15:15:47.616554 14192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3809eab701e47138431af019106ab8d" candidate_uuid: "ac66294fc8734840a60cbe11e4625260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2adbe25c005c4e93927b3d15df65f065"
I20260926 15:15:47.616724 14046 raft_consensus.cc:3037] T e3809eab701e47138431af019106ab8d P 6dc3ad57bb66483bbc89eb29244df045 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:47.616784 14192 raft_consensus.cc:3037] T e3809eab701e47138431af019106ab8d P 2adbe25c005c4e93927b3d15df65f065 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:47.618992 14046 raft_consensus.cc:2454] T e3809eab701e47138431af019106ab8d P 6dc3ad57bb66483bbc89eb29244df045 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac66294fc8734840a60cbe11e4625260 in term 1.
I20260926 15:15:47.619336 14263 leader_election.cc:302] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc3ad57bb66483bbc89eb29244df045, ac66294fc8734840a60cbe11e4625260; no voters:
I20260926 15:15:47.619460 14192 raft_consensus.cc:2454] T e3809eab701e47138431af019106ab8d P 2adbe25c005c4e93927b3d15df65f065 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac66294fc8734840a60cbe11e4625260 in term 1.
I20260926 15:15:47.619554 14414 raft_consensus.cc:2781] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:47.619940 14414 raft_consensus.cc:686] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260 [term 1 LEADER]: Becoming Leader. State: Replica: ac66294fc8734840a60cbe11e4625260, State: Running, Role: LEADER
I20260926 15:15:47.620369 14414 consensus_queue.cc:237] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adbe25c005c4e93927b3d15df65f065" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34425 } } peers { permanent_uuid: "ac66294fc8734840a60cbe11e4625260" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44185 } } peers { permanent_uuid: "6dc3ad57bb66483bbc89eb29244df045" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38531 } }
I20260926 15:15:47.624891 13117 catalog_manager.cc:5495] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac66294fc8734840a60cbe11e4625260 (127.6.209.195). New cstate: current_term: 1 leader_uuid: "ac66294fc8734840a60cbe11e4625260" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adbe25c005c4e93927b3d15df65f065" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac66294fc8734840a60cbe11e4625260" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dc3ad57bb66483bbc89eb29244df045" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38531 } health_report { overall_health: UNKNOWN } } }
I20260926 15:15:47.657924 14404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:47.653533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37774 (local address 127.6.209.195:44185)
0926 15:15:47.653697 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:47.653702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:47.653717 (+ 15us) server_negotiation.cc:407] Connection header received
0926 15:15:47.653815 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:47.653818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:47.653891 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:47.654005 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:15:47.654289 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.655038 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.656237 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.656581 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.656665 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:47.657051 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:15:47.657523 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:15:47.657626 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:47.657689 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 15:15:47.657741 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 15:15:47.665028 14046 raft_consensus.cc:1261] T e3809eab701e47138431af019106ab8d P 6dc3ad57bb66483bbc89eb29244df045 [term 1 FOLLOWER]: Refusing update from remote peer ac66294fc8734840a60cbe11e4625260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:47.665055 14192 raft_consensus.cc:1261] T e3809eab701e47138431af019106ab8d P 2adbe25c005c4e93927b3d15df65f065 [term 1 FOLLOWER]: Refusing update from remote peer ac66294fc8734840a60cbe11e4625260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:47.665860 14414 consensus_queue.cc:1035] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc3ad57bb66483bbc89eb29244df045" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:47.666100 14416 consensus_queue.cc:1035] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2adbe25c005c4e93927b3d15df65f065" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:47.674396 14419 mvcc.cc:205] Tried to move back new op lower bound from 7333624822427635712 to 7333624822256812032. Current Snapshot: MvccSnapshot[applied={T|T < 7333624822427635712}]
I20260926 15:15:47.681347 14420 mvcc.cc:205] Tried to move back new op lower bound from 7333624822427635712 to 7333624822256812032. Current Snapshot: MvccSnapshot[applied={T|T < 7333624822427635712}]
I20260926 15:15:47.693356 13117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:47.693648 13117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:47.703059 13117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 487b6fe6e7264e4db494c8057789998b: Sending DeleteTablet for 3 replicas of tablet e3809eab701e47138431af019106ab8d
I20260926 15:15:47.704051 14172 tablet_service.cc:1514] Processing DeleteTablet for tablet e3809eab701e47138431af019106ab8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:47 UTC) from {username='oryx', principal='oryx/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:36022
I20260926 15:15:47.704932 14024 tablet_service.cc:1514] Processing DeleteTablet for tablet e3809eab701e47138431af019106ab8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:47 UTC) from {username='oryx', principal='oryx/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:46810
I20260926 15:15:47.705425 14429 tablet_replica.cc:330] T e3809eab701e47138431af019106ab8d P 2adbe25c005c4e93927b3d15df65f065: stopping tablet replica
I20260926 15:15:47.705695 14430 tablet_replica.cc:330] T e3809eab701e47138431af019106ab8d P 6dc3ad57bb66483bbc89eb29244df045: stopping tablet replica
I20260926 15:15:47.705856 14429 raft_consensus.cc:2229] T e3809eab701e47138431af019106ab8d P 2adbe25c005c4e93927b3d15df65f065 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:47.705962 14310 tablet_service.cc:1514] Processing DeleteTablet for tablet e3809eab701e47138431af019106ab8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:47 UTC) from {username='oryx', principal='oryx/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:37772
I20260926 15:15:47.706044 14430 raft_consensus.cc:2229] T e3809eab701e47138431af019106ab8d P 6dc3ad57bb66483bbc89eb29244df045 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:47.706228 14429 raft_consensus.cc:2258] T e3809eab701e47138431af019106ab8d P 2adbe25c005c4e93927b3d15df65f065 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:47.706426 14430 raft_consensus.cc:2258] T e3809eab701e47138431af019106ab8d P 6dc3ad57bb66483bbc89eb29244df045 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:47.708159 14430 ts_tablet_manager.cc:1905] T e3809eab701e47138431af019106ab8d P 6dc3ad57bb66483bbc89eb29244df045: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:47.708159 14429 ts_tablet_manager.cc:1905] T e3809eab701e47138431af019106ab8d P 2adbe25c005c4e93927b3d15df65f065: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:47.715171 14429 ts_tablet_manager.cc:1918] T e3809eab701e47138431af019106ab8d P 2adbe25c005c4e93927b3d15df65f065: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:15:47.715310 14429 log.cc:1196] T e3809eab701e47138431af019106ab8d P 2adbe25c005c4e93927b3d15df65f065: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/e3809eab701e47138431af019106ab8d
I20260926 15:15:47.715363 14430 ts_tablet_manager.cc:1918] T e3809eab701e47138431af019106ab8d P 6dc3ad57bb66483bbc89eb29244df045: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:15:47.715497 14430 log.cc:1196] T e3809eab701e47138431af019106ab8d P 6dc3ad57bb66483bbc89eb29244df045: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/wal/wals/e3809eab701e47138431af019106ab8d
I20260926 15:15:47.715880 14429 ts_tablet_manager.cc:1939] T e3809eab701e47138431af019106ab8d P 2adbe25c005c4e93927b3d15df65f065: Deleting consensus metadata
I20260926 15:15:47.716059 14430 ts_tablet_manager.cc:1939] T e3809eab701e47138431af019106ab8d P 6dc3ad57bb66483bbc89eb29244df045: Deleting consensus metadata
I20260926 15:15:47.717469 13101 catalog_manager.cc:4841] TS 6dc3ad57bb66483bbc89eb29244df045 (127.6.209.193:38531): tablet e3809eab701e47138431af019106ab8d (table test-table [id=330397a910494126b441d5d4deafffef]) successfully deleted
I20260926 15:15:47.717708 13103 catalog_manager.cc:4841] TS 2adbe25c005c4e93927b3d15df65f065 (127.6.209.194:34425): tablet e3809eab701e47138431af019106ab8d (table test-table [id=330397a910494126b441d5d4deafffef]) successfully deleted
I20260926 15:15:47.721251 14431 tablet_replica.cc:330] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260: stopping tablet replica
I20260926 15:15:47.721580 14431 raft_consensus.cc:2229] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:15:47.722045 14431 raft_consensus.cc:2258] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:47.723459 14431 ts_tablet_manager.cc:1905] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:47.727404 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 13085
I20260926 15:15:47.729249 14431 ts_tablet_manager.cc:1918] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:15:47.729274 13147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:47.729400 14431 log.cc:1196] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/e3809eab701e47138431af019106ab8d
I20260926 15:15:47.729938 14431 ts_tablet_manager.cc:1939] T e3809eab701e47138431af019106ab8d P ac66294fc8734840a60cbe11e4625260: Deleting consensus metadata
I20260926 15:15:47.745870 14390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:47.709842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34814 (local address 127.6.209.253:38809)
0926 15:15:47.710096 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:47.710105 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:47.725780 (+ 15675us) server_negotiation.cc:407] Connection header received
0926 15:15:47.725980 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:47.725985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:47.726056 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:47.726137 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:47.728715 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.730953 (+ 2238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.734694 (+ 3741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.735340 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.735431 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:47.738909 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:47.738916 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:47.738921 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:47.738959 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:47.742249 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:47.743593 (+ 1344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:47.743598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:47.743602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:47.743758 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:47.744161 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:47.744166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:47.744169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:47.744515 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:47.744754 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:47.745337 (+ 583us) server_negotiation.cc:299] Negotiation successful
0926 15:15:47.745546 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 15:15:47.762185 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:47.726417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57146 (local address 127.6.209.252:40519)
0926 15:15:47.726603 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:47.726609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:47.746649 (+ 20040us) server_negotiation.cc:407] Connection header received
0926 15:15:47.746906 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:47.746912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:47.746987 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:47.747055 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:47.749048 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.750176 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.751972 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.752674 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.752786 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:47.756198 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:47.756204 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:47.756209 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:47.756245 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:47.759498 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:47.760201 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:47.760207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:47.760210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:47.760347 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:47.760705 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:47.760710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:47.760714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:47.761101 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:47.761328 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:47.761639 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 15:15:47.761864 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 15:15:47.846805 13104 catalog_manager.cc:4841] TS ac66294fc8734840a60cbe11e4625260 (127.6.209.195:44185): tablet e3809eab701e47138431af019106ab8d (table test-table [id=330397a910494126b441d5d4deafffef]) successfully deleted
W20260926 15:15:47.847028 13104 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.6.209.195:44185, user_credentials={real_user=slave}} blocked reactor thread for 115720us
I20260926 15:15:47.848954 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 13085
I20260926 15:15:47.904264 14390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:47.873305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34818 (local address 127.6.209.253:38809)
0926 15:15:47.873464 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:47.873470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:47.888651 (+ 15181us) server_negotiation.cc:407] Connection header received
0926 15:15:47.889718 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:47.889724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:47.889807 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:47.889869 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:47.891855 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.892843 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.894572 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.895193 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.895283 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:47.898437 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:47.898456 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:47.898461 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:47.898515 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:47.901716 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:47.902417 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:47.902423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:47.902426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:47.902561 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:47.902928 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:47.902933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:47.902936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:47.903246 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:47.903484 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:47.903727 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 15:15:47.903935 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 15:15:47.904620 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:47.881244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57154 (local address 127.6.209.252:40519)
0926 15:15:47.881462 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:47.881471 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:47.889175 (+ 7704us) server_negotiation.cc:407] Connection header received
0926 15:15:47.889470 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:47.889477 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:47.889569 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:47.889630 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:47.891482 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.892501 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.894242 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:47.894911 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:47.894993 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:47.898353 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:47.898358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:47.898364 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:47.898402 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:47.901682 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:47.902542 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:47.902547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:47.902550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:47.902688 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:47.903090 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:47.903095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:47.903098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:47.903433 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:47.903635 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:47.904101 (+ 466us) server_negotiation.cc:299] Negotiation successful
0926 15:15:47.904296 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 15:15:48.683723 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:48.682985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.6.209.193:41937)
0926 15:15:48.683414 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:48.683533 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":220,"threads_started":1}
W20260926 15:15:48.684221 14092 heartbeater.cc:644] Failed to heartbeat to 127.6.209.254:44331 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
I20260926 15:15:48.691334 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:48.690703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.6.209.195:56251)
0926 15:15:48.691072 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:48.691181 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":185,"threads_started":1}
W20260926 15:15:48.691629 14376 heartbeater.cc:644] Failed to heartbeat to 127.6.209.254:44331 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
I20260926 15:15:48.705170 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:48.704554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.6.209.194:45155)
0926 15:15:48.704954 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:48.705039 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":173,"threads_started":1}
W20260926 15:15:48.705543 14238 heartbeater.cc:644] Failed to heartbeat to 127.6.209.254:44331 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
I20260926 15:15:49.208381 14444 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 487b6fe6e7264e4db494c8057789998b)
I20260926 15:15:49.208598 14444 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: false } }
I20260926 15:15:49.209304 14444 leader_election.cc:288] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 487b6fe6e7264e4db494c8057789998b (127.6.209.254:44331), cd14dff0bba0414d97637969e5be9c5c (127.6.209.253:38809)
I20260926 15:15:49.210425 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.209347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.0.0.1:36620)
0926 15:15:49.209920 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.210178 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":312,"threads_started":1}
W20260926 15:15:49.210713 13898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
I20260926 15:15:49.212105 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.211803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.0.0.1:36626)
0926 15:15:49.211922 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.211997 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 15:15:49.212435 13898 leader_election.cc:334] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 487b6fe6e7264e4db494c8057789998b (127.6.209.254:44331): Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
I20260926 15:15:49.212639 14449 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 487b6fe6e7264e4db494c8057789998b)
I20260926 15:15:49.212869 14449 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: false } }
I20260926 15:15:49.214154 14449 leader_election.cc:288] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 487b6fe6e7264e4db494c8057789998b (127.6.209.254:44331), 47996a5d983a4193afbd0f1ee829e0f5 (127.6.209.252:40519)
I20260926 15:15:49.217113 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.209459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:38809 (local address 127.0.0.1:34824)
0926 15:15:49.210392 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.210404 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:49.210559 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:49.210887 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:49.210892 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:49.210963 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:49.211371 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:49.211383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.212676 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.212681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:49.214147 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:49.214163 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.216090 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.216098 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:49.216381 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:49.216433 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 15:15:49.216508 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 15:15:49.216677 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"thread_start_us":717,"threads_started":1}
I20260926 15:15:49.218446 14447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.209619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34824 (local address 127.6.209.253:38809)
0926 15:15:49.210180 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:49.210188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:49.210495 (+ 307us) server_negotiation.cc:407] Connection header received
0926 15:15:49.210697 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:49.210701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:49.210755 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:49.210857 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:49.211536 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.212496 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.214320 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.215831 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.217566 (+ 1735us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:49.217702 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:49.217761 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 15:15:49.217930 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":257,"threads_started":1}
I20260926 15:15:49.221705 13731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47996a5d983a4193afbd0f1ee829e0f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cd14dff0bba0414d97637969e5be9c5c" is_pre_election: true
I20260926 15:15:49.221901 14450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.214186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.0.0.1:36642)
0926 15:15:49.221327 (+ 7141us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.221533 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6968,"thread_start_us":6790,"threads_started":1}
I20260926 15:15:49.222025 13731 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47996a5d983a4193afbd0f1ee829e0f5 in term 1.
W20260926 15:15:49.222509 13707 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
I20260926 15:15:49.224131 13899 leader_election.cc:302] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47996a5d983a4193afbd0f1ee829e0f5, cd14dff0bba0414d97637969e5be9c5c; no voters: 487b6fe6e7264e4db494c8057789998b
I20260926 15:15:49.224438 14444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 15:15:49.224550 14444 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 487b6fe6e7264e4db494c8057789998b)
I20260926 15:15:49.224648 14444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [term 1 FOLLOWER]: Advancing to term 2
I20260926 15:15:49.224988 14451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.224414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.0.0.1:36648)
0926 15:15:49.224599 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.224749 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260926 15:15:49.225129 13707 leader_election.cc:334] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 487b6fe6e7264e4db494c8057789998b (127.6.209.254:44331): Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
I20260926 15:15:49.226753 14444 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: false } }
I20260926 15:15:49.227331 14444 leader_election.cc:288] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [CANDIDATE]: Term 2 election: Requested vote from peers 487b6fe6e7264e4db494c8057789998b (127.6.209.254:44331), cd14dff0bba0414d97637969e5be9c5c (127.6.209.253:38809)
I20260926 15:15:49.227747 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.227362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.0.0.1:36656)
0926 15:15:49.227528 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.227610 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 15:15:49.227782 13731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47996a5d983a4193afbd0f1ee829e0f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cd14dff0bba0414d97637969e5be9c5c"
I20260926 15:15:49.227985 13731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [term 1 FOLLOWER]: Advancing to term 2
I20260926 15:15:49.228565 14450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.214335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:40519 (local address 127.0.0.1:57166)
0926 15:15:49.222136 (+ 7801us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.222157 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:49.222333 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:49.223157 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:49.223164 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:49.223197 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:49.223489 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:49.223503 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.224747 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.224752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:49.226114 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:49.226124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.227266 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.227271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:49.227948 (+ 677us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:49.227987 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 15:15:49.228101 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 15:15:49.228226 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7688,"thread_start_us":8537,"threads_started":1}
I20260926 15:15:49.228605 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.214600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57166 (local address 127.6.209.252:40519)
0926 15:15:49.222896 (+ 8296us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:49.222902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:49.222920 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:15:49.222970 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:49.222974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:49.223035 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:49.223125 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:49.223635 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.224577 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.226261 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.227089 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.227214 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:49.227321 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:49.228146 (+ 825us) server_negotiation.cc:299] Negotiation successful
0926 15:15:49.228278 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8134,"thread_start_us":7974,"threads_started":1}
I20260926 15:15:49.228776 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.228334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.0.0.1:36668)
0926 15:15:49.228544 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.228634 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260926 15:15:49.229111 13898 leader_election.cc:334] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 487b6fe6e7264e4db494c8057789998b (127.6.209.254:44331): Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
I20260926 15:15:49.229352 13922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd14dff0bba0414d97637969e5be9c5c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "47996a5d983a4193afbd0f1ee829e0f5" is_pre_election: true
I20260926 15:15:49.229574 13922 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd14dff0bba0414d97637969e5be9c5c in current term 2: Already voted for candidate 47996a5d983a4193afbd0f1ee829e0f5 in this term.
I20260926 15:15:49.230070 13706 leader_election.cc:302] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd14dff0bba0414d97637969e5be9c5c; no voters: 47996a5d983a4193afbd0f1ee829e0f5, 487b6fe6e7264e4db494c8057789998b
I20260926 15:15:49.230386 13731 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47996a5d983a4193afbd0f1ee829e0f5 in term 2.
I20260926 15:15:49.230532 14449 raft_consensus.cc:2726] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 15:15:49.230734 13899 leader_election.cc:302] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47996a5d983a4193afbd0f1ee829e0f5, cd14dff0bba0414d97637969e5be9c5c; no voters: 487b6fe6e7264e4db494c8057789998b
I20260926 15:15:49.230939 14444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 15:15:49.231287 14444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [term 2 LEADER]: Becoming Leader. State: Replica: 47996a5d983a4193afbd0f1ee829e0f5, State: Running, Role: LEADER
I20260926 15:15:49.231585 14444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: false } }
I20260926 15:15:49.233264 14454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47996a5d983a4193afbd0f1ee829e0f5. Latest consensus state: current_term: 2 leader_uuid: "47996a5d983a4193afbd0f1ee829e0f5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: false } } }
I20260926 15:15:49.233542 14454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:49.234354 14456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:15:49.237816 14456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:15:49.238595 14456 catalog_manager.cc:1259] Loaded cluster ID: aaebb9b0e62d4ad5b164b4119414725a
I20260926 15:15:49.238655 14456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:15:49.239702 14456 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:15:49.240464 14456 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5: Loaded TSK: 0
I20260926 15:15:49.241305 14456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:15:49.262379 13912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:15:49.268296 13912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:15:49.271744 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.271369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.0.0.1:36682)
0926 15:15:49.271545 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.271615 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 15:15:49.271726 13731 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [term 2 FOLLOWER]: Refusing update from remote peer 47996a5d983a4193afbd0f1ee829e0f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 15:15:49.272308 14444 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 15:15:49.272565 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.272241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.0.0.1:36684)
0926 15:15:49.272366 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.272418 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 15:15:49.272768 13898 consensus_peers.cc:487] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 -> Peer 487b6fe6e7264e4db494c8057789998b (127.6.209.254:44331): Couldn't send request to peer 487b6fe6e7264e4db494c8057789998b. Status: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 15:15:49.274420 14449 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47996a5d983a4193afbd0f1ee829e0f5. Latest consensus state: current_term: 2 leader_uuid: "47996a5d983a4193afbd0f1ee829e0f5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: false } } }
I20260926 15:15:49.274663 14449 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:15:49.275875 14444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "47996a5d983a4193afbd0f1ee829e0f5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: false } } }
I20260926 15:15:49.276196 14444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:49.276600 14449 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "47996a5d983a4193afbd0f1ee829e0f5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "487b6fe6e7264e4db494c8057789998b" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44331 } } peers { permanent_uuid: "cd14dff0bba0414d97637969e5be9c5c" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 38809 } attrs { promote: false } } peers { permanent_uuid: "47996a5d983a4193afbd0f1ee829e0f5" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 40519 } attrs { promote: false } } }
I20260926 15:15:49.276803 14449 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd14dff0bba0414d97637969e5be9c5c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:15:49.288825 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.282640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:38531 (local address 127.0.0.1:46818)
0926 15:15:49.282843 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.282868 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:49.282950 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:49.284167 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:49.284171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:49.284194 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:49.284355 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:49.284365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.285615 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.285622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:49.286817 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:49.286830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.288100 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.288106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:49.288304 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:49.288326 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 15:15:49.288426 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 15:15:49.288549 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260926 15:15:49.289005 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.282746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46818 (local address 127.6.209.193:38531)
0926 15:15:49.283881 (+ 1135us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:49.283886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:49.283902 (+ 16us) server_negotiation.cc:407] Connection header received
0926 15:15:49.283958 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:49.283963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:49.284037 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:49.284483 (+ 446us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:49.284633 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.285486 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.286987 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.287940 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.288077 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:49.288174 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:49.288449 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 15:15:49.288585 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":983,"thread_start_us":909,"threads_started":1}
I20260926 15:15:49.289682 14024 tablet_service.cc:1467] Processing CreateTablet for tablet f9f7a40d56d54f95b2483efcb78e0406 (DEFAULT_TABLE table=test-table [id=70788e0ae3d240c2a30cf28e0da61118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:49.290052 14024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9f7a40d56d54f95b2483efcb78e0406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:49.296428 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.283342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:34425 (local address 127.0.0.1:36024)
0926 15:15:49.289008 (+ 5666us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.289025 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:49.289118 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:49.289478 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:49.289482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:49.289501 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:49.289635 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:49.289643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.291758 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.291762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:49.293333 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:49.293346 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.295515 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.295519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:49.295790 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:49.295834 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 15:15:49.295903 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 15:15:49.296055 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5595,"thread_start_us":12723,"threads_started":1}
I20260926 15:15:49.297418 14466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.283674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36024 (local address 127.6.209.194:34425)
0926 15:15:49.284975 (+ 1301us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:49.284998 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:49.289207 (+ 4209us) server_negotiation.cc:407] Connection header received
0926 15:15:49.289276 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:49.289282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:49.289356 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:49.289758 (+ 402us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:49.289879 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.291602 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.293509 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.295343 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.296650 (+ 1307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:49.296782 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:49.296850 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 15:15:49.297023 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1189,"thread_start_us":1076,"threads_started":1}
I20260926 15:15:49.298316 14172 tablet_service.cc:1467] Processing CreateTablet for tablet f9f7a40d56d54f95b2483efcb78e0406 (DEFAULT_TABLE table=test-table [id=70788e0ae3d240c2a30cf28e0da61118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:49.298887 14172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9f7a40d56d54f95b2483efcb78e0406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:49.300446 14467 tablet_bootstrap.cc:492] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045: Bootstrap starting.
I20260926 15:15:49.301251 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.283084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:44185 (local address 127.0.0.1:37782)
0926 15:15:49.283319 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.283337 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:49.283451 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:49.296366 (+ 12915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:49.296369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:49.296388 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:49.296637 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:49.296645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.297719 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.297724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:49.299326 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:49.299336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.300531 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.300537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:49.300702 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:49.300722 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 15:15:49.300847 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 15:15:49.300993 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260926 15:15:49.301299 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.283395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37782 (local address 127.6.209.195:44185)
0926 15:15:49.295650 (+ 12255us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:49.295658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:49.295678 (+ 20us) server_negotiation.cc:407] Connection header received
0926 15:15:49.295753 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:49.295759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:49.295844 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:49.296356 (+ 512us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:49.296760 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.297463 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.299500 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.300344 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.300486 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:49.300584 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:49.300886 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 15:15:49.301010 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12120,"thread_start_us":12815,"threads_started":1}
I20260926 15:15:49.301892 14310 tablet_service.cc:1467] Processing CreateTablet for tablet f9f7a40d56d54f95b2483efcb78e0406 (DEFAULT_TABLE table=test-table [id=70788e0ae3d240c2a30cf28e0da61118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:49.302251 14310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9f7a40d56d54f95b2483efcb78e0406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:49.302676 14467 tablet_bootstrap.cc:654] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:49.304688 14467 tablet_bootstrap.cc:492] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045: No bootstrap required, opened a new log
I20260926 15:15:49.304832 14467 ts_tablet_manager.cc:1397] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 15:15:49.306006 14467 raft_consensus.cc:348] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc3ad57bb66483bbc89eb29244df045" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38531 } } peers { permanent_uuid: "ac66294fc8734840a60cbe11e4625260" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44185 } } peers { permanent_uuid: "2adbe25c005c4e93927b3d15df65f065" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34425 } }
I20260926 15:15:49.306324 14467 raft_consensus.cc:374] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:49.306438 14467 raft_consensus.cc:729] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc3ad57bb66483bbc89eb29244df045, State: Initialized, Role: FOLLOWER
I20260926 15:15:49.306807 14467 consensus_queue.cc:260] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc3ad57bb66483bbc89eb29244df045" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38531 } } peers { permanent_uuid: "ac66294fc8734840a60cbe11e4625260" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44185 } } peers { permanent_uuid: "2adbe25c005c4e93927b3d15df65f065" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34425 } }
I20260926 15:15:49.307868 14467 ts_tablet_manager.cc:1428] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 15:15:49.310724 14091 heartbeater.cc:503] Master 127.6.209.252:40519 was elected leader, sending a full tablet report...
I20260926 15:15:49.316205 14469 tablet_bootstrap.cc:492] T f9f7a40d56d54f95b2483efcb78e0406 P 2adbe25c005c4e93927b3d15df65f065: Bootstrap starting.
I20260926 15:15:49.316983 14471 tablet_bootstrap.cc:492] T f9f7a40d56d54f95b2483efcb78e0406 P ac66294fc8734840a60cbe11e4625260: Bootstrap starting.
I20260926 15:15:49.317480 14472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.308685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.6.209.193:37617)
0926 15:15:49.317136 (+ 8451us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.317258 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8334,"threads_started":1}
I20260926 15:15:49.318230 14469 tablet_bootstrap.cc:654] T f9f7a40d56d54f95b2483efcb78e0406 P 2adbe25c005c4e93927b3d15df65f065: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:49.318522 14471 tablet_bootstrap.cc:654] T f9f7a40d56d54f95b2483efcb78e0406 P ac66294fc8734840a60cbe11e4625260: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:49.320199 14472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.319874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.6.209.193:35239)
0926 15:15:49.320002 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.320070 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 15:15:49.321442 14471 tablet_bootstrap.cc:492] T f9f7a40d56d54f95b2483efcb78e0406 P ac66294fc8734840a60cbe11e4625260: No bootstrap required, opened a new log
I20260926 15:15:49.321545 14471 ts_tablet_manager.cc:1397] T f9f7a40d56d54f95b2483efcb78e0406 P ac66294fc8734840a60cbe11e4625260: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260926 15:15:49.322090 14471 raft_consensus.cc:348] T f9f7a40d56d54f95b2483efcb78e0406 P ac66294fc8734840a60cbe11e4625260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc3ad57bb66483bbc89eb29244df045" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38531 } } peers { permanent_uuid: "ac66294fc8734840a60cbe11e4625260" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44185 } } peers { permanent_uuid: "2adbe25c005c4e93927b3d15df65f065" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34425 } }
I20260926 15:15:49.322242 14471 raft_consensus.cc:374] T f9f7a40d56d54f95b2483efcb78e0406 P ac66294fc8734840a60cbe11e4625260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:49.322299 14471 raft_consensus.cc:729] T f9f7a40d56d54f95b2483efcb78e0406 P ac66294fc8734840a60cbe11e4625260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac66294fc8734840a60cbe11e4625260, State: Initialized, Role: FOLLOWER
I20260926 15:15:49.322481 14471 consensus_queue.cc:260] T f9f7a40d56d54f95b2483efcb78e0406 P ac66294fc8734840a60cbe11e4625260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc3ad57bb66483bbc89eb29244df045" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38531 } } peers { permanent_uuid: "ac66294fc8734840a60cbe11e4625260" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44185 } } peers { permanent_uuid: "2adbe25c005c4e93927b3d15df65f065" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34425 } }
I20260926 15:15:49.323148 14471 ts_tablet_manager.cc:1428] T f9f7a40d56d54f95b2483efcb78e0406 P ac66294fc8734840a60cbe11e4625260: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 15:15:49.324146 14469 tablet_bootstrap.cc:492] T f9f7a40d56d54f95b2483efcb78e0406 P 2adbe25c005c4e93927b3d15df65f065: No bootstrap required, opened a new log
I20260926 15:15:49.324273 14469 ts_tablet_manager.cc:1397] T f9f7a40d56d54f95b2483efcb78e0406 P 2adbe25c005c4e93927b3d15df65f065: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 15:15:49.324795 14469 raft_consensus.cc:348] T f9f7a40d56d54f95b2483efcb78e0406 P 2adbe25c005c4e93927b3d15df65f065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc3ad57bb66483bbc89eb29244df045" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38531 } } peers { permanent_uuid: "ac66294fc8734840a60cbe11e4625260" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44185 } } peers { permanent_uuid: "2adbe25c005c4e93927b3d15df65f065" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34425 } }
I20260926 15:15:49.325030 14469 raft_consensus.cc:374] T f9f7a40d56d54f95b2483efcb78e0406 P 2adbe25c005c4e93927b3d15df65f065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:49.325122 14469 raft_consensus.cc:729] T f9f7a40d56d54f95b2483efcb78e0406 P 2adbe25c005c4e93927b3d15df65f065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2adbe25c005c4e93927b3d15df65f065, State: Initialized, Role: FOLLOWER
I20260926 15:15:49.325330 14469 consensus_queue.cc:260] T f9f7a40d56d54f95b2483efcb78e0406 P 2adbe25c005c4e93927b3d15df65f065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc3ad57bb66483bbc89eb29244df045" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38531 } } peers { permanent_uuid: "ac66294fc8734840a60cbe11e4625260" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44185 } } peers { permanent_uuid: "2adbe25c005c4e93927b3d15df65f065" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34425 } }
I20260926 15:15:49.326882 14375 heartbeater.cc:503] Master 127.6.209.252:40519 was elected leader, sending a full tablet report...
I20260926 15:15:49.327476 14477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.326493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.6.209.195:54633)
0926 15:15:49.327035 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.327191 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"threads_started":1}
I20260926 15:15:49.328537 14469 ts_tablet_manager.cc:1428] T f9f7a40d56d54f95b2483efcb78e0406 P 2adbe25c005c4e93927b3d15df65f065: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260926 15:15:49.328872 14477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.328467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.6.209.195:39483)
0926 15:15:49.328678 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.328745 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260926 15:15:49.330379 14478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.329058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.6.209.194:36299)
0926 15:15:49.329972 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.330120 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":784,"thread_start_us":390,"threads_started":1}
I20260926 15:15:49.330474 14237 heartbeater.cc:503] Master 127.6.209.252:40519 was elected leader, sending a full tablet report...
I20260926 15:15:49.332336 14479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.330948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.6.209.194:35999)
0926 15:15:49.332039 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.332161 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":982,"thread_start_us":850,"threads_started":1}
I20260926 15:15:49.556373 14470 raft_consensus.cc:482] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:15:49.556579 14470 raft_consensus.cc:504] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc3ad57bb66483bbc89eb29244df045" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38531 } } peers { permanent_uuid: "ac66294fc8734840a60cbe11e4625260" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44185 } } peers { permanent_uuid: "2adbe25c005c4e93927b3d15df65f065" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34425 } }
I20260926 15:15:49.558040 14470 leader_election.cc:288] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac66294fc8734840a60cbe11e4625260 (127.6.209.195:44185), 2adbe25c005c4e93927b3d15df65f065 (127.6.209.194:34425)
I20260926 15:15:49.562937 14466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.558515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:34423 (local address 127.6.209.194:34425)
0926 15:15:49.558729 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:49.558732 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:49.558747 (+ 15us) server_negotiation.cc:407] Connection header received
0926 15:15:49.558804 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:49.558807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:49.558854 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:49.558949 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:49.559373 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.559969 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.561512 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.562183 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.562336 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:49.562398 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:49.562619 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 15:15:49.562726 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 15:15:49.562961 14472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.558262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:34425 (local address 127.6.209.193:34423)
0926 15:15:49.558424 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.558442 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:49.558558 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:49.558973 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:49.558978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:49.559019 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:49.559251 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:49.559258 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.560086 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.560091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:49.561372 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:49.561383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.562337 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.562342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:49.562487 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:49.562506 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 15:15:49.562595 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 15:15:49.562706 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 15:15:49.563409 14192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f7a40d56d54f95b2483efcb78e0406" candidate_uuid: "6dc3ad57bb66483bbc89eb29244df045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2adbe25c005c4e93927b3d15df65f065" is_pre_election: true
I20260926 15:15:49.563560 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.558262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:44185 (local address 127.6.209.193:38917)
0926 15:15:49.559161 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.559172 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:49.559294 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:49.559575 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:49.559579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:49.559596 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:49.559847 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:49.559854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.560656 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.560661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:49.561963 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:49.561974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.562925 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.562930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:49.563122 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:49.563146 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 15:15:49.563218 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 15:15:49.563325 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":405,"thread_start_us":262,"threads_started":1}
I20260926 15:15:49.563575 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.558415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:38917 (local address 127.6.209.195:44185)
0926 15:15:49.558624 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:49.558647 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:49.559260 (+ 613us) server_negotiation.cc:407] Connection header received
0926 15:15:49.559428 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:49.559432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:49.559481 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:49.559563 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:49.559968 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.560534 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.562106 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.562745 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.562906 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:49.562968 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:49.563260 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 15:15:49.563366 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 15:15:49.563618 14192 raft_consensus.cc:2454] T f9f7a40d56d54f95b2483efcb78e0406 P 2adbe25c005c4e93927b3d15df65f065 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc3ad57bb66483bbc89eb29244df045 in term 0.
I20260926 15:15:49.564025 13979 leader_election.cc:302] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2adbe25c005c4e93927b3d15df65f065, 6dc3ad57bb66483bbc89eb29244df045; no voters:
I20260926 15:15:49.564329 14470 raft_consensus.cc:2781] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:15:49.564436 14470 raft_consensus.cc:482] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:15:49.564278 14330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f7a40d56d54f95b2483efcb78e0406" candidate_uuid: "6dc3ad57bb66483bbc89eb29244df045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac66294fc8734840a60cbe11e4625260" is_pre_election: true
I20260926 15:15:49.564519 14470 raft_consensus.cc:3037] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:49.564639 14330 raft_consensus.cc:2454] T f9f7a40d56d54f95b2483efcb78e0406 P ac66294fc8734840a60cbe11e4625260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc3ad57bb66483bbc89eb29244df045 in term 0.
I20260926 15:15:49.566430 14470 raft_consensus.cc:504] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc3ad57bb66483bbc89eb29244df045" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38531 } } peers { permanent_uuid: "ac66294fc8734840a60cbe11e4625260" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44185 } } peers { permanent_uuid: "2adbe25c005c4e93927b3d15df65f065" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34425 } }
I20260926 15:15:49.566864 14470 leader_election.cc:288] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045 [CANDIDATE]: Term 1 election: Requested vote from peers ac66294fc8734840a60cbe11e4625260 (127.6.209.195:44185), 2adbe25c005c4e93927b3d15df65f065 (127.6.209.194:34425)
I20260926 15:15:49.567269 14330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f7a40d56d54f95b2483efcb78e0406" candidate_uuid: "6dc3ad57bb66483bbc89eb29244df045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac66294fc8734840a60cbe11e4625260"
I20260926 15:15:49.567266 14192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f7a40d56d54f95b2483efcb78e0406" candidate_uuid: "6dc3ad57bb66483bbc89eb29244df045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2adbe25c005c4e93927b3d15df65f065"
I20260926 15:15:49.567413 14330 raft_consensus.cc:3037] T f9f7a40d56d54f95b2483efcb78e0406 P ac66294fc8734840a60cbe11e4625260 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:49.567480 14192 raft_consensus.cc:3037] T f9f7a40d56d54f95b2483efcb78e0406 P 2adbe25c005c4e93927b3d15df65f065 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:49.569542 14192 raft_consensus.cc:2454] T f9f7a40d56d54f95b2483efcb78e0406 P 2adbe25c005c4e93927b3d15df65f065 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc3ad57bb66483bbc89eb29244df045 in term 1.
I20260926 15:15:49.569639 14330 raft_consensus.cc:2454] T f9f7a40d56d54f95b2483efcb78e0406 P ac66294fc8734840a60cbe11e4625260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc3ad57bb66483bbc89eb29244df045 in term 1.
I20260926 15:15:49.569855 13979 leader_election.cc:302] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2adbe25c005c4e93927b3d15df65f065, 6dc3ad57bb66483bbc89eb29244df045; no voters:
I20260926 15:15:49.570078 14470 raft_consensus.cc:2781] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:49.570631 14470 raft_consensus.cc:686] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045 [term 1 LEADER]: Becoming Leader. State: Replica: 6dc3ad57bb66483bbc89eb29244df045, State: Running, Role: LEADER
I20260926 15:15:49.571055 14470 consensus_queue.cc:237] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc3ad57bb66483bbc89eb29244df045" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38531 } } peers { permanent_uuid: "ac66294fc8734840a60cbe11e4625260" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44185 } } peers { permanent_uuid: "2adbe25c005c4e93927b3d15df65f065" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34425 } }
I20260926 15:15:49.571434 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.571089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.6.209.193:50655)
0926 15:15:49.571220 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.571317 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 15:15:49.573873 13911 catalog_manager.cc:5495] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dc3ad57bb66483bbc89eb29244df045 (127.6.209.193). New cstate: current_term: 1 leader_uuid: "6dc3ad57bb66483bbc89eb29244df045" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dc3ad57bb66483bbc89eb29244df045" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac66294fc8734840a60cbe11e4625260" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2adbe25c005c4e93927b3d15df65f065" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34425 } health_report { overall_health: UNKNOWN } } }
I20260926 15:15:49.651832 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.647016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46828 (local address 127.6.209.193:38531)
0926 15:15:49.647259 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:49.647263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:49.647279 (+ 16us) server_negotiation.cc:407] Connection header received
0926 15:15:49.647482 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:49.647486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:49.647557 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:49.647675 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:15:49.648031 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.648794 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.650026 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.650394 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.650474 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:49.650825 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:15:49.651285 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:15:49.651392 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:49.651492 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 15:15:49.651543 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 15:15:49.655503 14192 raft_consensus.cc:1261] T f9f7a40d56d54f95b2483efcb78e0406 P 2adbe25c005c4e93927b3d15df65f065 [term 1 FOLLOWER]: Refusing update from remote peer 6dc3ad57bb66483bbc89eb29244df045: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:49.655856 14330 raft_consensus.cc:1261] T f9f7a40d56d54f95b2483efcb78e0406 P ac66294fc8734840a60cbe11e4625260 [term 1 FOLLOWER]: Refusing update from remote peer 6dc3ad57bb66483bbc89eb29244df045: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:49.656185 14481 consensus_queue.cc:1035] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2adbe25c005c4e93927b3d15df65f065" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:49.656615 14470 consensus_queue.cc:1035] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac66294fc8734840a60cbe11e4625260" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:49.660171 14477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.659477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.6.209.195:59877)
0926 15:15:49.659840 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.659956 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 15:15:49.660557 14479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.660164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.6.209.194:52817)
0926 15:15:49.660309 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.660411 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 15:15:49.666123 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.665630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.6.209.193:36741)
0926 15:15:49.665872 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.665972 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260926 15:15:49.668045 14477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.667564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.6.209.195:36639)
0926 15:15:49.667808 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.667906 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260926 15:15:49.669010 14479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.668520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.6.209.194:59625)
0926 15:15:49.668756 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.668871 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260926 15:15:49.677731 13912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:49.677958 13912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:49.682569 13912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47996a5d983a4193afbd0f1ee829e0f5: Sending DeleteTablet for 3 replicas of tablet f9f7a40d56d54f95b2483efcb78e0406
I20260926 15:15:49.683486 14024 tablet_service.cc:1514] Processing DeleteTablet for tablet f9f7a40d56d54f95b2483efcb78e0406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:49 UTC) from {username='oryx', principal='oryx/127.6.209.252@KRBTEST.COM'} at 127.0.0.1:46818
I20260926 15:15:49.683635 14310 tablet_service.cc:1514] Processing DeleteTablet for tablet f9f7a40d56d54f95b2483efcb78e0406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:49 UTC) from {username='oryx', principal='oryx/127.6.209.252@KRBTEST.COM'} at 127.0.0.1:37782
I20260926 15:15:49.684306 14172 tablet_service.cc:1514] Processing DeleteTablet for tablet f9f7a40d56d54f95b2483efcb78e0406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:49 UTC) from {username='oryx', principal='oryx/127.6.209.252@KRBTEST.COM'} at 127.0.0.1:36024
I20260926 15:15:49.685137 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 13885
I20260926 15:15:49.688215 14494 tablet_replica.cc:330] T f9f7a40d56d54f95b2483efcb78e0406 P ac66294fc8734840a60cbe11e4625260: stopping tablet replica
I20260926 15:15:49.688407 14494 raft_consensus.cc:2229] T f9f7a40d56d54f95b2483efcb78e0406 P ac66294fc8734840a60cbe11e4625260 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:49.688651 14494 raft_consensus.cc:2258] T f9f7a40d56d54f95b2483efcb78e0406 P ac66294fc8734840a60cbe11e4625260 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:49.689563 14494 ts_tablet_manager.cc:1905] T f9f7a40d56d54f95b2483efcb78e0406 P ac66294fc8734840a60cbe11e4625260: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:49.694072 14494 ts_tablet_manager.cc:1918] T f9f7a40d56d54f95b2483efcb78e0406 P ac66294fc8734840a60cbe11e4625260: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:15:49.694226 14494 log.cc:1196] T f9f7a40d56d54f95b2483efcb78e0406 P ac66294fc8734840a60cbe11e4625260: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/f9f7a40d56d54f95b2483efcb78e0406
I20260926 15:15:49.694990 14494 ts_tablet_manager.cc:1939] T f9f7a40d56d54f95b2483efcb78e0406 P ac66294fc8734840a60cbe11e4625260: Deleting consensus metadata
I20260926 15:15:49.696413 13899 catalog_manager.cc:4841] TS ac66294fc8734840a60cbe11e4625260 (127.6.209.195:44185): tablet f9f7a40d56d54f95b2483efcb78e0406 (table test-table [id=70788e0ae3d240c2a30cf28e0da61118]) successfully deleted
I20260926 15:15:49.697537 14495 tablet_replica.cc:330] T f9f7a40d56d54f95b2483efcb78e0406 P 2adbe25c005c4e93927b3d15df65f065: stopping tablet replica
I20260926 15:15:49.697755 14495 raft_consensus.cc:2229] T f9f7a40d56d54f95b2483efcb78e0406 P 2adbe25c005c4e93927b3d15df65f065 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:49.698056 14495 raft_consensus.cc:2258] T f9f7a40d56d54f95b2483efcb78e0406 P 2adbe25c005c4e93927b3d15df65f065 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:49.698647 14493 tablet_replica.cc:330] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045: stopping tablet replica
I20260926 15:15:49.698859 14493 raft_consensus.cc:2229] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:15:49.699447 14493 raft_consensus.cc:2258] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:49.700466 14493 ts_tablet_manager.cc:1905] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:49.700701 14495 ts_tablet_manager.cc:1905] T f9f7a40d56d54f95b2483efcb78e0406 P 2adbe25c005c4e93927b3d15df65f065: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:49.706679 14495 ts_tablet_manager.cc:1918] T f9f7a40d56d54f95b2483efcb78e0406 P 2adbe25c005c4e93927b3d15df65f065: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:15:49.706722 14493 ts_tablet_manager.cc:1918] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:15:49.706801 14495 log.cc:1196] T f9f7a40d56d54f95b2483efcb78e0406 P 2adbe25c005c4e93927b3d15df65f065: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/f9f7a40d56d54f95b2483efcb78e0406
I20260926 15:15:49.706954 14493 log.cc:1196] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435694426955-6983-0/minicluster-data/ts-0/wal/wals/f9f7a40d56d54f95b2483efcb78e0406
I20260926 15:15:49.707261 14495 ts_tablet_manager.cc:1939] T f9f7a40d56d54f95b2483efcb78e0406 P 2adbe25c005c4e93927b3d15df65f065: Deleting consensus metadata
I20260926 15:15:49.707687 14493 ts_tablet_manager.cc:1939] T f9f7a40d56d54f95b2483efcb78e0406 P 6dc3ad57bb66483bbc89eb29244df045: Deleting consensus metadata
I20260926 15:15:49.708684 13898 catalog_manager.cc:4841] TS 2adbe25c005c4e93927b3d15df65f065 (127.6.209.194:34425): tablet f9f7a40d56d54f95b2483efcb78e0406 (table test-table [id=70788e0ae3d240c2a30cf28e0da61118]) successfully deleted
I20260926 15:15:49.708886 13896 catalog_manager.cc:4841] TS 6dc3ad57bb66483bbc89eb29244df045 (127.6.209.193:38531): tablet f9f7a40d56d54f95b2483efcb78e0406 (table test-table [id=70788e0ae3d240c2a30cf28e0da61118]) successfully deleted
I20260926 15:15:49.713645 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.686230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46834 (local address 127.6.209.193:38531)
0926 15:15:49.686685 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:49.686694 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:49.697285 (+ 10591us) server_negotiation.cc:407] Connection header received
0926 15:15:49.697574 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:49.697580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:49.697660 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:49.697735 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:49.699699 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.700747 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.702581 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.703220 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.703309 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:49.706736 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:49.706751 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:49.706765 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:49.706808 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:49.710725 (+ 3917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:49.711321 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:49.711330 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:49.711334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:49.711514 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:49.711891 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:49.711897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:49.711903 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:49.712235 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:49.712498 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:49.713019 (+ 521us) server_negotiation.cc:299] Negotiation successful
0926 15:15:49.713217 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305}
I20260926 15:15:49.714275 14086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:49.805572 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 13885
I20260926 15:15:49.823457 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 14102
I20260926 15:15:49.835412 14466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.824140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36032 (local address 127.6.209.194:34425)
0926 15:15:49.824274 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:49.824278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:49.824294 (+ 16us) server_negotiation.cc:407] Connection header received
0926 15:15:49.824456 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:49.824459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:49.824510 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:49.824583 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:49.825864 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.826513 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.827642 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.828002 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.828085 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:49.830471 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:49.830483 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:49.830498 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:49.830526 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:49.833029 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:49.833652 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:49.833659 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:49.833661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:49.833782 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:49.834152 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:49.834155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:49.834157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:49.834504 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:49.834688 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:49.834976 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 15:15:49.835098 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 15:15:49.835915 14232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:49.842268 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.841700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.0.0.1:36700)
0926 15:15:49.842059 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.842133 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":176,"threads_started":1}
I20260926 15:15:49.843385 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.843114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44331 (local address 127.0.0.1:36712)
0926 15:15:49.843219 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:49.843279 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 15:15:49.922608 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 14102
I20260926 15:15:49.938083 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 14247
I20260926 15:15:49.950382 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:49.938761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37796 (local address 127.6.209.195:44185)
0926 15:15:49.938915 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:49.938919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:49.938935 (+ 16us) server_negotiation.cc:407] Connection header received
0926 15:15:49.939051 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:49.939054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:49.939107 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:49.939181 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:49.940331 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.941075 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.942193 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:49.942531 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:49.942606 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:49.945275 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:49.945288 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:49.945307 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:49.945347 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:49.948300 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:49.948885 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:49.948892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:49.948894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:49.949018 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:49.949296 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:49.949299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:49.949300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:49.949577 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:49.949753 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:49.949928 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 15:15:49.950049 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 15:15:49.950974 14370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:50.058439 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 14247
I20260926 15:15:50.077059 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 13577
I20260926 15:15:50.077980 13751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:50.183616 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 13577
I20260926 15:15:50.201565 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 13676
I20260926 15:15:50.202428 13942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:50.307310 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 13676
2026-09-26T15:15:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8849 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:15:50 dist-test-slave-j6q7 krb5kdc[14506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:15:50 dist-test-slave-j6q7 krb5kdc[14506](info): set up 2 sockets
Sep 26 15:15:50 dist-test-slave-j6q7 krb5kdc[14506](info): commencing operation
krb5kdc: starting...
W20260926 15:15:51.309646 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:15:51 dist-test-slave-j6q7 krb5kdc[14506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:15:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:15:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.254.keytab.
Entry for principal oryx/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.254.keytab.
I20260926 15:15:51.480407 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:38245
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.254.keytab
--principal=oryx/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:42993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:38245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:51.763557 14519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:51.763825 14519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:51.763967 14519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:51.770284 14519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:15:51.770411 14519 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 15:15:51.770524 14519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:51.770612 14519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:51.770686 14519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:15:51.770757 14519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:15:51.778842 14519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:42993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:38245
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.254.keytab
--principal=oryx/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:38245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:51.779846 14519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:51.781024 14519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:51.791486 14527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:51.791577 14524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:51.791543 14525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:51.792202 14519 server_base.cc:1034] running on GCE node
I20260926 15:15:51.792932 14519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:51.794100 14519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:51.795423 14519 hybrid_clock.cc:648] HybridClock initialized: now 1790435751795384 us; error 39 us; skew 500 ppm
I20260926 15:15:51.795714 14519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:51 dist-test-slave-j6q7 krb5kdc[14506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435751, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:51.798532 14519 init.cc:377] Logged in from keytab as oryx/127.6.209.254@KRBTEST.COM (short username oryx)
I20260926 15:15:51.799734 14519 webserver.cc:460] Webserver started at http://127.6.209.254:46253/ using document root <none> and password file <none>
I20260926 15:15:51.800325 14519 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:51.800403 14519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:51.800643 14519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:51.803278 14519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "d00a0cbd2c3049aab7d7e3d29d5568e9"
format_stamp: "Formatted at 2026-09-26 15:15:51 on dist-test-slave-j6q7"
server_key: "f685ded71f2fe340b83bd62e56dea387"
server_key_iv: "5c6ec641eba5fcb0a3ba3dc993d67b9f"
server_key_version: "encryptionkey@0"
I20260926 15:15:51.803900 14519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "d00a0cbd2c3049aab7d7e3d29d5568e9"
format_stamp: "Formatted at 2026-09-26 15:15:51 on dist-test-slave-j6q7"
server_key: "f685ded71f2fe340b83bd62e56dea387"
server_key_iv: "5c6ec641eba5fcb0a3ba3dc993d67b9f"
server_key_version: "encryptionkey@0"
I20260926 15:15:51.808133 14519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 15:15:51.811092 14534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:51.812105 14519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:15:51.812305 14519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "d00a0cbd2c3049aab7d7e3d29d5568e9"
format_stamp: "Formatted at 2026-09-26 15:15:51 on dist-test-slave-j6q7"
server_key: "f685ded71f2fe340b83bd62e56dea387"
server_key_iv: "5c6ec641eba5fcb0a3ba3dc993d67b9f"
server_key_version: "encryptionkey@0"
I20260926 15:15:51.812508 14519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:51.823899 14519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:51.827029 14519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:51.827252 14519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:51.846318 14519 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:38245
I20260926 15:15:51.846342 14595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:38245 every 8 connection(s)
I20260926 15:15:51.847401 14519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:15:51.850569 14596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:51.854257 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 14519
I20260926 15:15:51.854471 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:15:51.855130 6983 external_mini_cluster.cc:1428] Setting key dcaff4fd3505c96a9211fc047cf489ad
I20260926 15:15:51.861123 14596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9: Bootstrap starting.
I20260926 15:15:51.864132 14596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:51.865103 14596 log.cc:824] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:51.867424 14596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9: No bootstrap required, opened a new log
Sep 26 15:15:51 dist-test-slave-j6q7 krb5kdc[14506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM
I20260926 15:15:51.876511 14596 raft_consensus.cc:348] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00a0cbd2c3049aab7d7e3d29d5568e9" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38245 } }
I20260926 15:15:51.876827 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:51.856884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42790 (local address 127.6.209.254:38245)
0926 15:15:51.857672 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:51.857703 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:51.857749 (+ 46us) server_negotiation.cc:407] Connection header received
0926 15:15:51.859991 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:51.860014 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:51.860399 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:51.860743 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:51.861916 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:51.863591 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:51.864769 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:51.865736 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:51.865853 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:51.869590 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:51.869609 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:51.869626 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:51.869669 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:51.873140 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:51.873672 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:51.873679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:51.873685 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:51.873889 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:51.874182 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:51.874187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:51.874190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:51.874796 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:51.875057 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:51.875312 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 15:15:51.875675 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":315,"threads_started":1}
I20260926 15:15:51.876936 14596 raft_consensus.cc:374] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:51.877074 14596 raft_consensus.cc:729] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00a0cbd2c3049aab7d7e3d29d5568e9, State: Initialized, Role: FOLLOWER
I20260926 15:15:51.877674 14596 consensus_queue.cc:260] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00a0cbd2c3049aab7d7e3d29d5568e9" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38245 } }
I20260926 15:15:51.877902 14596 raft_consensus.cc:388] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:15:51.878041 14596 raft_consensus.cc:482] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:15:51.878230 14596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:51.880416 14596 raft_consensus.cc:504] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00a0cbd2c3049aab7d7e3d29d5568e9" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38245 } }
I20260926 15:15:51.880786 14596 leader_election.cc:302] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d00a0cbd2c3049aab7d7e3d29d5568e9; no voters:
I20260926 15:15:51.881330 14596 leader_election.cc:288] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:15:51.881477 14601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:51.882174 14601 raft_consensus.cc:686] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9 [term 1 LEADER]: Becoming Leader. State: Replica: d00a0cbd2c3049aab7d7e3d29d5568e9, State: Running, Role: LEADER
I20260926 15:15:51.882699 14601 consensus_queue.cc:237] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00a0cbd2c3049aab7d7e3d29d5568e9" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38245 } }
I20260926 15:15:51.883044 14596 sys_catalog.cc:564] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:15:51.886236 14602 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d00a0cbd2c3049aab7d7e3d29d5568e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00a0cbd2c3049aab7d7e3d29d5568e9" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38245 } } }
I20260926 15:15:51.886554 14602 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:51.886981 14603 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d00a0cbd2c3049aab7d7e3d29d5568e9. Latest consensus state: current_term: 1 leader_uuid: "d00a0cbd2c3049aab7d7e3d29d5568e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00a0cbd2c3049aab7d7e3d29d5568e9" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38245 } } }
I20260926 15:15:51.887241 14603 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9 [sys.catalog]: This master's current role is: LEADER
W20260926 15:15:51.893348 14617 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:15:51.893452 14617 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:15:51.893568 14612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:15:51.894955 14612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:15:51.901357 14612 catalog_manager.cc:1347] Generated new cluster ID: b2cb94487490421f816d7f553b29cde6
I20260926 15:15:51.901448 14612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:15:51.915122 14612 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:15:51.916441 14612 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:15:51.925101 14612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9: Generated new TSK 0
I20260926 15:15:51.925889 14612 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.193.keytab.
Entry for principal oryx/127.6.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.193.keytab.
I20260926 15:15:51.968122 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:38245
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.193.keytab
--principal=oryx/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:42993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:52.251561 14622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:52.251775 14622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:52.251852 14622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:52.257886 14622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:52.257995 14622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:52.258138 14622 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:15:52.258266 14622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:15:52.266336 14622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:42993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.193.keytab
--principal=oryx/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.209.254:38245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:52.267316 14622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:52.268469 14622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:52.279541 14630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:52.279515 14628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:52.279945 14627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:52.280303 14622 server_base.cc:1034] running on GCE node
I20260926 15:15:52.280895 14622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:52.281725 14622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:52.282913 14622 hybrid_clock.cc:648] HybridClock initialized: now 1790435752282864 us; error 37 us; skew 500 ppm
I20260926 15:15:52.283154 14622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:52 dist-test-slave-j6q7 krb5kdc[14506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435752, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:52.286052 14622 init.cc:377] Logged in from keytab as oryx/127.6.209.193@KRBTEST.COM (short username oryx)
I20260926 15:15:52.287289 14622 webserver.cc:460] Webserver started at http://127.6.209.193:36413/ using document root <none> and password file <none>
I20260926 15:15:52.287876 14622 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:52.287982 14622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:52.288288 14622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:52.290787 14622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e"
format_stamp: "Formatted at 2026-09-26 15:15:52 on dist-test-slave-j6q7"
server_key: "8b865b4f5e8340121b1ee4ce23b80906"
server_key_iv: "f1d441e91703f8fe15d52bdc211487ca"
server_key_version: "encryptionkey@0"
I20260926 15:15:52.291433 14622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e"
format_stamp: "Formatted at 2026-09-26 15:15:52 on dist-test-slave-j6q7"
server_key: "8b865b4f5e8340121b1ee4ce23b80906"
server_key_iv: "f1d441e91703f8fe15d52bdc211487ca"
server_key_version: "encryptionkey@0"
I20260926 15:15:52.295646 14622 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 15:15:52.298456 14637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:52.299512 14622 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 15:15:52.299657 14622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e"
format_stamp: "Formatted at 2026-09-26 15:15:52 on dist-test-slave-j6q7"
server_key: "8b865b4f5e8340121b1ee4ce23b80906"
server_key_iv: "f1d441e91703f8fe15d52bdc211487ca"
server_key_version: "encryptionkey@0"
I20260926 15:15:52.299844 14622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:52.334554 14622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:52.338080 14622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:52.338342 14622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:52.340483 14622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:52.340571 14622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:52.340674 14622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:52.340754 14622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 15:15:52 dist-test-slave-j6q7 krb5kdc[14506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435752, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM
I20260926 15:15:52.359830 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:52.341037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:50817 (local address 127.6.209.254:38245)
0926 15:15:52.341226 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:52.341234 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:52.342500 (+ 1266us) server_negotiation.cc:407] Connection header received
0926 15:15:52.344148 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:52.344153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:52.344237 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:52.344328 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:52.346157 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:52.346983 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:52.349490 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:52.350025 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:52.350127 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:52.354187 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:52.354194 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:52.354197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:52.354242 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:52.356663 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:52.357439 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:52.357445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:52.357448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:52.357590 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:52.358162 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:52.358168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:52.358171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:52.358557 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:52.358717 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:52.359314 (+ 597us) server_negotiation.cc:299] Negotiation successful
0926 15:15:52.359516 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 15:15:52.360754 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:52.341623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:38245 (local address 127.6.209.193:50817)
0926 15:15:52.342367 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:52.342399 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:52.343921 (+ 1522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:52.344544 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:52.344557 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:52.345279 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:52.346001 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:52.346020 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:52.347173 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:52.347179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:52.349348 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:52.349361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:52.350227 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:52.350234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:52.350418 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:52.351059 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:52.351079 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:52.353930 (+ 2851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:52.356781 (+ 2851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:52.356788 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:52.356800 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:52.357298 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:52.357747 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:52.357753 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:52.357758 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:52.358035 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:52.358692 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:52.358700 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:52.359022 (+ 322us) client_negotiation.cc:769] Sending connection context
0926 15:15:52.359228 (+ 206us) client_negotiation.cc:240] Negotiation successful
0926 15:15:52.359651 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":343,"threads_started":1}
I20260926 15:15:52.378006 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:52.365643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:48353 (local address 127.6.209.254:38245)
0926 15:15:52.365806 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:52.365811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:52.366621 (+ 810us) server_negotiation.cc:407] Connection header received
0926 15:15:52.366798 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:52.366801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:52.366843 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:52.366922 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:52.367626 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:52.368296 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:52.369887 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:52.370290 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:52.370405 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:52.372627 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:52.372631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:52.372634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:52.372656 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:52.375304 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:52.376103 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:52.376110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:52.376115 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:52.376256 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:52.376669 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:52.376675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:52.376679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:52.376983 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:52.377115 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:52.377468 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 15:15:52.377646 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 15:15:52.378110 14701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:52.365897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:38245 (local address 127.6.209.193:48353)
0926 15:15:52.366523 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:52.366537 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:52.366671 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:52.366939 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:52.366943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:52.367296 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:52.367514 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:52.367521 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:52.368427 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:52.368432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:52.369758 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:52.369768 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:52.370480 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:52.370488 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:52.370722 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:52.371256 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:52.371282 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:52.372428 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:52.375460 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:52.375465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:52.375468 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:52.375994 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:52.376371 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:52.376375 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:52.376380 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:52.376539 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:52.377100 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:52.377105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:52.377257 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 15:15:52.377382 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 15:15:52.377632 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":335,"threads_started":1}
I20260926 15:15:52.389472 14622 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:40547
I20260926 15:15:52.389482 14756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:40547 every 8 connection(s)
I20260926 15:15:52.390450 14622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:15:52.392195 14757 heartbeater.cc:347] Connected to a master server at 127.6.209.254:38245
I20260926 15:15:52.392359 14757 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:52.392983 14757 heartbeater.cc:511] Master 127.6.209.254:38245 requested a full tablet report, sending...
I20260926 15:15:52.393151 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 14622
I20260926 15:15:52.393355 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:15:52.393975 6983 external_mini_cluster.cc:1428] Setting key a1ac716574a96a383134cee40992232c
I20260926 15:15:52.394753 14551 ts_manager.cc:194] Registered new tserver with Master: 92e8f5d7dc5949468ab368fdfd2b2d2e (127.6.209.193:40547)
I20260926 15:15:52.396039 14551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.209.193@KRBTEST.COM'} at 127.6.209.193:50817
WARNING: no policy specified for oryx/127.6.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.194.keytab.
Entry for principal oryx/127.6.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.194.keytab.
I20260926 15:15:52.433419 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:38245
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.194.keytab
--principal=oryx/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:42993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:52.719494 14761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:52.719714 14761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:52.719872 14761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:52.726022 14761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:52.726119 14761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:52.726239 14761 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:15:52.726351 14761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:15:52.735001 14761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:42993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.194.keytab
--principal=oryx/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.209.254:38245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:52.736034 14761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:52.737237 14761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:52.748925 14766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:52.748888 14767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:52.749028 14769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:52.749699 14761 server_base.cc:1034] running on GCE node
I20260926 15:15:52.750197 14761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:52.750931 14761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:52.752140 14761 hybrid_clock.cc:648] HybridClock initialized: now 1790435752752086 us; error 39 us; skew 500 ppm
I20260926 15:15:52.752475 14761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:52 dist-test-slave-j6q7 krb5kdc[14506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435752, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:52.755151 14761 init.cc:377] Logged in from keytab as oryx/127.6.209.194@KRBTEST.COM (short username oryx)
I20260926 15:15:52.756297 14761 webserver.cc:460] Webserver started at http://127.6.209.194:37957/ using document root <none> and password file <none>
I20260926 15:15:52.756906 14761 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:52.757006 14761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:52.757294 14761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:52.759720 14761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "b6b7fefcc60a4f6fae92bc19687153a0"
format_stamp: "Formatted at 2026-09-26 15:15:52 on dist-test-slave-j6q7"
server_key: "15b69012d9dedeecf2a652d0f3472252"
server_key_iv: "1535b8d6564be155d91c85f06873467e"
server_key_version: "encryptionkey@0"
I20260926 15:15:52.760391 14761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "b6b7fefcc60a4f6fae92bc19687153a0"
format_stamp: "Formatted at 2026-09-26 15:15:52 on dist-test-slave-j6q7"
server_key: "15b69012d9dedeecf2a652d0f3472252"
server_key_iv: "1535b8d6564be155d91c85f06873467e"
server_key_version: "encryptionkey@0"
I20260926 15:15:52.764734 14761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 15:15:52.767666 14776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:52.768851 14761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260926 15:15:52.769024 14761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "b6b7fefcc60a4f6fae92bc19687153a0"
format_stamp: "Formatted at 2026-09-26 15:15:52 on dist-test-slave-j6q7"
server_key: "15b69012d9dedeecf2a652d0f3472252"
server_key_iv: "1535b8d6564be155d91c85f06873467e"
server_key_version: "encryptionkey@0"
I20260926 15:15:52.769212 14761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:52.790530 14761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:52.793767 14761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:52.793994 14761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:52.796254 14761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:52.796350 14761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:52.796471 14761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:52.796563 14761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 26 15:15:52 dist-test-slave-j6q7 krb5kdc[14506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435752, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM
I20260926 15:15:52.817494 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:52.796585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:54497 (local address 127.6.209.254:38245)
0926 15:15:52.796773 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:52.796779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:52.798159 (+ 1380us) server_negotiation.cc:407] Connection header received
0926 15:15:52.799944 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:52.799949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:52.800028 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:52.800126 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:52.802209 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:52.803376 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:52.805649 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:52.806277 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:52.806401 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:52.810352 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:52.810361 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:52.810368 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:52.810416 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:52.813745 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:52.814688 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:52.814695 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:52.814709 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:52.814871 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:52.815267 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:52.815271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:52.815275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:52.815633 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:52.815787 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:52.816980 (+ 1193us) server_negotiation.cc:299] Negotiation successful
0926 15:15:52.817158 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 15:15:52.818167 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:52.797146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:38245 (local address 127.6.209.194:54497)
0926 15:15:52.797986 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:52.798038 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:52.799696 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:52.800323 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:52.800332 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:52.801052 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:52.801915 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:52.801958 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:52.803520 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:52.803529 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:52.805477 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:52.805509 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:52.806512 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:52.806521 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:52.806810 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:52.807551 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:52.807584 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:52.810132 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:52.813878 (+ 3746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:52.813889 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:52.813904 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:52.814535 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:52.814976 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:52.814979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:52.814981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:52.815142 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:52.815768 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:52.815774 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:52.816227 (+ 453us) client_negotiation.cc:769] Sending connection context
0926 15:15:52.816523 (+ 296us) client_negotiation.cc:240] Negotiation successful
0926 15:15:52.817025 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":369,"threads_started":1}
I20260926 15:15:52.835696 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:52.822936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:34021 (local address 127.6.209.254:38245)
0926 15:15:52.823065 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:52.823068 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:52.823640 (+ 572us) server_negotiation.cc:407] Connection header received
0926 15:15:52.823799 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:52.823803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:52.823858 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:52.823945 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:52.824706 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:52.825539 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:52.827175 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:52.827562 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:52.827691 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:52.829619 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:52.829623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:52.829626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:52.829648 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:52.833065 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:52.833908 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:52.833913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:52.833917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:52.834068 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:52.834409 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:52.834415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:52.834418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:52.834669 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:52.834810 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:52.835174 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 15:15:52.835376 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 15:15:52.835881 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:52.823007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:38245 (local address 127.6.209.194:34021)
0926 15:15:52.823550 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:52.823566 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:52.823688 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:52.823964 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:52.823968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:52.824353 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:52.824586 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:52.824593 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:52.825659 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:52.825664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:52.827021 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:52.827031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:52.827734 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:52.827758 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:52.827918 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:52.828458 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:52.828473 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:52.829423 (+ 950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:52.833206 (+ 3783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:52.833212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:52.833217 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:52.833730 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:52.834170 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:52.834176 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:52.834183 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:52.834307 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:52.834784 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:52.834789 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:52.834950 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 15:15:52.835083 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 15:15:52.835425 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":301,"threads_started":1}
I20260926 15:15:52.845403 14761 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:37611
I20260926 15:15:52.845422 14895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:37611 every 8 connection(s)
I20260926 15:15:52.846639 14761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:15:52.848397 14896 heartbeater.cc:347] Connected to a master server at 127.6.209.254:38245
I20260926 15:15:52.848614 14896 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:52.848645 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 14761
I20260926 15:15:52.848829 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:15:52.849257 14896 heartbeater.cc:511] Master 127.6.209.254:38245 requested a full tablet report, sending...
I20260926 15:15:52.849282 6983 external_mini_cluster.cc:1428] Setting key 3f9cba38f3f4f4c6d88c78fad96d0878
I20260926 15:15:52.850519 14550 ts_manager.cc:194] Registered new tserver with Master: b6b7fefcc60a4f6fae92bc19687153a0 (127.6.209.194:37611)
I20260926 15:15:52.851614 14550 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.209.194@KRBTEST.COM'} at 127.6.209.194:54497
WARNING: no policy specified for oryx/127.6.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.195.keytab.
Entry for principal oryx/127.6.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.195.keytab.
I20260926 15:15:52.891270 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:38245
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.195.keytab
--principal=oryx/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:42993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:53.177676 14900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:53.177937 14900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:53.178103 14900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:53.184569 14900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:53.184662 14900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:53.184798 14900 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:15:53.184922 14900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:15:53.193349 14900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:42993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.195.keytab
--principal=oryx/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.209.254:38245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:53.194336 14900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:53.195422 14900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:53.207355 14905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:53.207448 14906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:53.207388 14908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:53.210041 14900 server_base.cc:1034] running on GCE node
I20260926 15:15:53.210538 14900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:53.211362 14900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:53.212602 14900 hybrid_clock.cc:648] HybridClock initialized: now 1790435753212548 us; error 44 us; skew 500 ppm
I20260926 15:15:53.212939 14900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:53 dist-test-slave-j6q7 krb5kdc[14506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435753, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:53.216019 14900 init.cc:377] Logged in from keytab as oryx/127.6.209.195@KRBTEST.COM (short username oryx)
I20260926 15:15:53.217425 14900 webserver.cc:460] Webserver started at http://127.6.209.195:43011/ using document root <none> and password file <none>
I20260926 15:15:53.218042 14900 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:53.218155 14900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:53.218392 14900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:53.221127 14900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "5274e26d9ee742fc9d5f93465ff450e8"
format_stamp: "Formatted at 2026-09-26 15:15:53 on dist-test-slave-j6q7"
server_key: "3487b9122e47c7e430bc7727e28667be"
server_key_iv: "b4b910474dff6961b78083d345b2b0f6"
server_key_version: "encryptionkey@0"
I20260926 15:15:53.221815 14900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "5274e26d9ee742fc9d5f93465ff450e8"
format_stamp: "Formatted at 2026-09-26 15:15:53 on dist-test-slave-j6q7"
server_key: "3487b9122e47c7e430bc7727e28667be"
server_key_iv: "b4b910474dff6961b78083d345b2b0f6"
server_key_version: "encryptionkey@0"
I20260926 15:15:53.226181 14900 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:15:53.229110 14915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:53.230319 14900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:15:53.230434 14900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "5274e26d9ee742fc9d5f93465ff450e8"
format_stamp: "Formatted at 2026-09-26 15:15:53 on dist-test-slave-j6q7"
server_key: "3487b9122e47c7e430bc7727e28667be"
server_key_iv: "b4b910474dff6961b78083d345b2b0f6"
server_key_version: "encryptionkey@0"
I20260926 15:15:53.230589 14900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:53.251026 14900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:53.254216 14900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:53.254397 14900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:53.256597 14900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:53.256708 14900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:53.256898 14900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:53.256976 14900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 15:15:53 dist-test-slave-j6q7 krb5kdc[14506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435753, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM
I20260926 15:15:53.279227 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:53.256914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:36353 (local address 127.6.209.254:38245)
0926 15:15:53.257077 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:53.257083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:53.258492 (+ 1409us) server_negotiation.cc:407] Connection header received
0926 15:15:53.260340 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:53.260347 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:53.260435 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:53.260560 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:53.262575 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:53.263740 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:53.266025 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:53.266404 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:53.266517 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:53.271539 (+ 5022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:53.271545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:53.271550 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:53.271591 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:53.275605 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:53.276548 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:53.276556 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:53.276562 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:53.276753 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:53.277329 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:53.277334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:53.277338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:53.277680 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:53.277863 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:53.278669 (+ 806us) server_negotiation.cc:299] Negotiation successful
0926 15:15:53.278888 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 15:15:53.280025 14924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:53.257437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:38245 (local address 127.6.209.195:36353)
0926 15:15:53.258312 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:53.258362 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:53.260097 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:53.260888 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:53.260898 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:53.261615 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:53.262327 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:53.262350 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:53.263913 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:53.263919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:53.265905 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:53.265914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:53.266590 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:53.266596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:53.266789 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:53.267362 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:53.267393 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:53.271236 (+ 3843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:53.275773 (+ 4537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:53.275783 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:53.275800 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:53.276416 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:53.276971 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:53.276975 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:53.276981 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:53.277204 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:53.277837 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:53.277844 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:53.278314 (+ 470us) client_negotiation.cc:769] Sending connection context
0926 15:15:53.278580 (+ 266us) client_negotiation.cc:240] Negotiation successful
0926 15:15:53.278993 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":410,"threads_started":1}
I20260926 15:15:53.299224 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:53.285159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:56521 (local address 127.6.209.254:38245)
0926 15:15:53.285322 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:53.285329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:53.286797 (+ 1468us) server_negotiation.cc:407] Connection header received
0926 15:15:53.286872 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:53.286879 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:53.286969 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:53.287090 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:53.287918 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:53.289248 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:53.291146 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:53.291740 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:53.291870 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:53.293790 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:53.293796 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:53.293800 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:53.293839 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:53.296867 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:53.297391 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:53.297398 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:53.297401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:53.297520 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:53.297874 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:53.297881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:53.297884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:53.298120 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:53.298277 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:53.298664 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 15:15:53.298847 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 15:15:53.299306 14985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:53.285356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:38245 (local address 127.6.209.195:56521)
0926 15:15:53.286622 (+ 1266us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:53.286636 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:53.286776 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:53.287097 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:53.287100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:53.287521 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:53.287782 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:53.287790 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:53.289387 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:53.289394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:53.290963 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:53.290982 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:53.291905 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:53.291910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:53.292061 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:53.292658 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:53.292673 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:53.293634 (+ 961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:53.296969 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:53.296973 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:53.296977 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:53.297292 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:53.297613 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:53.297619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:53.297622 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:53.297734 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:53.298247 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:53.298252 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:53.298410 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 15:15:53.298542 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 15:15:53.298836 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1160,"thread_start_us":1029,"threads_started":1}
I20260926 15:15:53.308706 14900 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:36427
I20260926 15:15:53.308746 15034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:36427 every 8 connection(s)
I20260926 15:15:53.309760 14900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:15:53.311398 15035 heartbeater.cc:347] Connected to a master server at 127.6.209.254:38245
I20260926 15:15:53.311545 15035 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:53.311986 15035 heartbeater.cc:511] Master 127.6.209.254:38245 requested a full tablet report, sending...
I20260926 15:15:53.313043 14550 ts_manager.cc:194] Registered new tserver with Master: 5274e26d9ee742fc9d5f93465ff450e8 (127.6.209.195:36427)
I20260926 15:15:53.313716 14550 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.209.195@KRBTEST.COM'} at 127.6.209.195:36353
I20260926 15:15:53.317309 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 14900
I20260926 15:15:53.317471 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:15:53.317870 6983 external_mini_cluster.cc:1428] Setting key 1ead9338046dedce1a965d0dc8ac4d94
I20260926 15:15:53.320379 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 15:15:53 dist-test-slave-j6q7 krb5kdc[14506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435753, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 15:15:53 dist-test-slave-j6q7 krb5kdc[14506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:15:53 dist-test-slave-j6q7 krb5kdc[14506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:15:53.398330 14757 heartbeater.cc:503] Master 127.6.209.254:38245 was elected leader, sending a full tablet report...
I20260926 15:15:53.854328 14896 heartbeater.cc:503] Master 127.6.209.254:38245 was elected leader, sending a full tablet report...
I20260926 15:15:54.315647 15035 heartbeater.cc:503] Master 127.6.209.254:38245 was elected leader, sending a full tablet report...
W20260926 15:15:54.340468 15043 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.983s user 0.000s sys 0.012s
W20260926 15:15:54.341467 14599 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:15:53.352128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42800 (local address 127.6.209.254:38245)
0926 15:15:53.352317 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:53.352323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:53.352547 (+ 224us) server_negotiation.cc:407] Connection header received
0926 15:15:53.352709 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:53.352713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:53.352780 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:53.352888 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:53.354000 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:53.354617 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:53.355723 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:53.356260 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:53.356410 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.341283 (+984873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42800: BlockingRecv error: recv got EOF from 127.0.0.1:42800 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 15:15:54 dist-test-slave-j6q7 krb5kdc[14506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435753, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM
I20260926 15:15:54.364545 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.345562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42802 (local address 127.6.209.254:38245)
0926 15:15:54.345775 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:54.345781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:54.346028 (+ 247us) server_negotiation.cc:407] Connection header received
0926 15:15:54.346190 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:54.346193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:54.346254 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:54.346331 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:54.347602 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.348296 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.349606 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.349956 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.350040 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.359856 (+ 9816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:54.359861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:54.359864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:54.359891 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:54.362403 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:54.362949 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:54.362952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:54.362954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:54.363090 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:54.363402 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:54.363405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:54.363407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:54.363664 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:54.363825 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:54.364049 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 15:15:54.364223 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 15:15:54.367798 14550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:15:54.374204 14550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:15:54.407225 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.385649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46760 (local address 127.6.209.195:36427)
0926 15:15:54.394939 (+ 9290us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:54.394950 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:54.401077 (+ 6127us) server_negotiation.cc:407] Connection header received
0926 15:15:54.401280 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:54.401291 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:54.401486 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:54.401658 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:54.402185 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.403151 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.405469 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.406269 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.406416 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.406594 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:54.406833 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 15:15:54.406961 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9143,"thread_start_us":9018,"threads_started":1}
I20260926 15:15:54.407254 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.385199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:36427 (local address 127.0.0.1:46760)
0926 15:15:54.400968 (+ 15769us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:54.400984 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:54.401108 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:54.401657 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:54.401661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:54.401684 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:54.402052 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:54.402062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.403374 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.403379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:54.405339 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:54.405349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.406462 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.406468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:54.406624 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.406675 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 15:15:54.406792 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 15:15:54.406936 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15671,"mutex_wait_us":2,"thread_start_us":16016,"threads_started":1}
I20260926 15:15:54.409662 15055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.384655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:40547 (local address 127.0.0.1:56352)
0926 15:15:54.397825 (+ 13170us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:54.397858 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:54.398019 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:54.398777 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:54.398789 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:54.398828 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:54.399313 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:54.399326 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.403491 (+ 4165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.403499 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:54.406602 (+ 3103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:54.406618 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.408999 (+ 2381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.409007 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:54.409183 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.409209 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 15:15:54.409283 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 15:15:54.409399 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13016,"thread_start_us":13467,"threads_started":1}
I20260926 15:15:54.410254 15058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.385377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56352 (local address 127.6.209.193:40547)
0926 15:15:54.393096 (+ 7719us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:54.393104 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:54.398002 (+ 4898us) server_negotiation.cc:407] Connection header received
0926 15:15:54.398208 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:54.398219 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:54.398522 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:54.398751 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:54.399512 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.403293 (+ 3781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.406805 (+ 3512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.408218 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.408339 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.408530 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:54.409893 (+ 1363us) server_negotiation.cc:299] Negotiation successful
0926 15:15:54.410030 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7572,"thread_start_us":8718,"threads_started":1}
I20260926 15:15:54.410375 14965 tablet_service.cc:1467] Processing CreateTablet for tablet fe174fe1178644bba2df4e50c0cb17f0 (DEFAULT_TABLE table=test-table [id=f7d835c929354ce3870ef84fa3d4728b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:54.411760 14965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe174fe1178644bba2df4e50c0cb17f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:54.412755 14687 tablet_service.cc:1467] Processing CreateTablet for tablet fe174fe1178644bba2df4e50c0cb17f0 (DEFAULT_TABLE table=test-table [id=f7d835c929354ce3870ef84fa3d4728b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:54.413877 14687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe174fe1178644bba2df4e50c0cb17f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:54.414806 15054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.384549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:37611 (local address 127.0.0.1:56226)
0926 15:15:54.395940 (+ 11391us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:54.395992 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:54.396159 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:54.408477 (+ 12318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:54.408481 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:54.408506 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:54.408872 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:54.408887 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.410612 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.410617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:54.412353 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:54.412364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.414167 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.414174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:54.414336 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.414366 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 15:15:54.414446 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 15:15:54.414549 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11043,"thread_start_us":11798,"threads_started":1}
I20260926 15:15:54.414877 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.384701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56226 (local address 127.6.209.194:37611)
0926 15:15:54.407769 (+ 23068us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:54.407776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:54.407799 (+ 23us) server_negotiation.cc:407] Connection header received
0926 15:15:54.407873 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:54.407879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:54.408101 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:54.409084 (+ 983us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:54.409263 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.410490 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.412496 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.413979 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.414099 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.414254 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:54.414495 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 15:15:54.414622 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22928,"thread_start_us":23541,"threads_started":1}
I20260926 15:15:54.417798 14826 tablet_service.cc:1467] Processing CreateTablet for tablet fe174fe1178644bba2df4e50c0cb17f0 (DEFAULT_TABLE table=test-table [id=f7d835c929354ce3870ef84fa3d4728b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:54.419332 14826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe174fe1178644bba2df4e50c0cb17f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:54.422722 15060 tablet_bootstrap.cc:492] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8: Bootstrap starting.
I20260926 15:15:54.425392 15060 tablet_bootstrap.cc:654] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:54.426700 15060 log.cc:824] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:54.428916 15061 tablet_bootstrap.cc:492] T fe174fe1178644bba2df4e50c0cb17f0 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Bootstrap starting.
I20260926 15:15:54.429215 15060 tablet_bootstrap.cc:492] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8: No bootstrap required, opened a new log
I20260926 15:15:54.429427 15060 ts_tablet_manager.cc:1397] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 15:15:54.431751 15061 tablet_bootstrap.cc:654] T fe174fe1178644bba2df4e50c0cb17f0 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:54.431900 15063 tablet_bootstrap.cc:492] T fe174fe1178644bba2df4e50c0cb17f0 P b6b7fefcc60a4f6fae92bc19687153a0: Bootstrap starting.
I20260926 15:15:54.432926 15061 log.cc:824] T fe174fe1178644bba2df4e50c0cb17f0 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:54.435101 15063 tablet_bootstrap.cc:654] T fe174fe1178644bba2df4e50c0cb17f0 P b6b7fefcc60a4f6fae92bc19687153a0: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:54.435611 15061 tablet_bootstrap.cc:492] T fe174fe1178644bba2df4e50c0cb17f0 P 92e8f5d7dc5949468ab368fdfd2b2d2e: No bootstrap required, opened a new log
I20260926 15:15:54.435832 15061 ts_tablet_manager.cc:1397] T fe174fe1178644bba2df4e50c0cb17f0 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 15:15:54.436136 15063 log.cc:824] T fe174fe1178644bba2df4e50c0cb17f0 P b6b7fefcc60a4f6fae92bc19687153a0: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:54.437506 15060 raft_consensus.cc:348] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.437858 15060 raft_consensus.cc:374] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:54.438002 15060 raft_consensus.cc:729] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5274e26d9ee742fc9d5f93465ff450e8, State: Initialized, Role: FOLLOWER
I20260926 15:15:54.438678 15063 tablet_bootstrap.cc:492] T fe174fe1178644bba2df4e50c0cb17f0 P b6b7fefcc60a4f6fae92bc19687153a0: No bootstrap required, opened a new log
I20260926 15:15:54.438575 15060 consensus_queue.cc:260] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.438930 15063 ts_tablet_manager.cc:1397] T fe174fe1178644bba2df4e50c0cb17f0 P b6b7fefcc60a4f6fae92bc19687153a0: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 15:15:54.440094 15060 ts_tablet_manager.cc:1428] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260926 15:15:54.443493 15061 raft_consensus.cc:348] T fe174fe1178644bba2df4e50c0cb17f0 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.443840 15061 raft_consensus.cc:374] T fe174fe1178644bba2df4e50c0cb17f0 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:54.443943 15061 raft_consensus.cc:729] T fe174fe1178644bba2df4e50c0cb17f0 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e8f5d7dc5949468ab368fdfd2b2d2e, State: Initialized, Role: FOLLOWER
I20260926 15:15:54.444595 15061 consensus_queue.cc:260] T fe174fe1178644bba2df4e50c0cb17f0 P 92e8f5d7dc5949468ab368fdfd2b2d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.445976 15061 ts_tablet_manager.cc:1428] T fe174fe1178644bba2df4e50c0cb17f0 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260926 15:15:54.446846 15063 raft_consensus.cc:348] T fe174fe1178644bba2df4e50c0cb17f0 P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.447156 15063 raft_consensus.cc:374] T fe174fe1178644bba2df4e50c0cb17f0 P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:54.447254 15063 raft_consensus.cc:729] T fe174fe1178644bba2df4e50c0cb17f0 P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b7fefcc60a4f6fae92bc19687153a0, State: Initialized, Role: FOLLOWER
I20260926 15:15:54.447650 15063 consensus_queue.cc:260] T fe174fe1178644bba2df4e50c0cb17f0 P b6b7fefcc60a4f6fae92bc19687153a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.449141 15063 ts_tablet_manager.cc:1428] T fe174fe1178644bba2df4e50c0cb17f0 P b6b7fefcc60a4f6fae92bc19687153a0: Time spent starting tablet: real 0.010s user 0.004s sys 0.006s
I20260926 15:15:54.461824 15066 raft_consensus.cc:482] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:15:54.462044 15066 raft_consensus.cc:504] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.463330 15066 leader_election.cc:288] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6b7fefcc60a4f6fae92bc19687153a0 (127.6.209.194:37611), 92e8f5d7dc5949468ab368fdfd2b2d2e (127.6.209.193:40547)
I20260926 15:15:54.470930 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.463643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:34239 (local address 127.6.209.194:37611)
0926 15:15:54.463795 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:54.463801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:54.464386 (+ 585us) server_negotiation.cc:407] Connection header received
0926 15:15:54.464532 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:54.464537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:54.464603 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:54.464684 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:54.465417 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.466328 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.468902 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.469969 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.470107 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.470165 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:54.470636 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 15:15:54.470743 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 15:15:54.470930 15058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.463643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:53311 (local address 127.6.209.193:40547)
0926 15:15:54.463795 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:54.463801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:54.464386 (+ 585us) server_negotiation.cc:407] Connection header received
0926 15:15:54.464533 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:54.464537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:54.464603 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:54.464684 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:54.465440 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.466331 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.468902 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.469968 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.470106 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.470165 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:54.470636 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 15:15:54.470737 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 15:15:54.471132 15069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.463559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:37611 (local address 127.6.209.195:34239)
0926 15:15:54.464262 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:54.464285 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:54.464414 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:54.464709 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:54.464713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:54.464747 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:54.465287 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:54.465303 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.466439 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.466442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:54.468740 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:54.468747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.470217 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.470221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:54.470411 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.470526 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 15:15:54.470603 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 15:15:54.470758 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":321,"spinlock_wait_cycles":13312,"thread_start_us":406,"threads_started":1}
I20260926 15:15:54.471240 15070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.463688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:40547 (local address 127.6.209.195:53311)
0926 15:15:54.464300 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:54.464316 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:54.464434 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:54.464709 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:54.464713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:54.464749 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:54.465311 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:54.465318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.466440 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.466444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:54.468733 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:54.468747 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.470193 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.470202 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:54.470412 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.470525 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 15:15:54.470604 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 15:15:54.470758 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":224,"spinlock_wait_cycles":27008,"thread_start_us":313,"threads_started":1}
I20260926 15:15:54.471987 14712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe174fe1178644bba2df4e50c0cb17f0" candidate_uuid: "5274e26d9ee742fc9d5f93465ff450e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" is_pre_election: true
I20260926 15:15:54.472007 14851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe174fe1178644bba2df4e50c0cb17f0" candidate_uuid: "5274e26d9ee742fc9d5f93465ff450e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" is_pre_election: true
I20260926 15:15:54.472342 14712 raft_consensus.cc:2454] T fe174fe1178644bba2df4e50c0cb17f0 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5274e26d9ee742fc9d5f93465ff450e8 in term 0.
I20260926 15:15:54.472370 14851 raft_consensus.cc:2454] T fe174fe1178644bba2df4e50c0cb17f0 P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5274e26d9ee742fc9d5f93465ff450e8 in term 0.
I20260926 15:15:54.472908 14916 leader_election.cc:302] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5274e26d9ee742fc9d5f93465ff450e8, 92e8f5d7dc5949468ab368fdfd2b2d2e; no voters:
I20260926 15:15:54.473291 15066 raft_consensus.cc:2781] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:15:54.473413 15066 raft_consensus.cc:482] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:15:54.473481 15066 raft_consensus.cc:3037] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:54.475195 15066 raft_consensus.cc:504] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.475592 15066 leader_election.cc:288] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8 [CANDIDATE]: Term 1 election: Requested vote from peers b6b7fefcc60a4f6fae92bc19687153a0 (127.6.209.194:37611), 92e8f5d7dc5949468ab368fdfd2b2d2e (127.6.209.193:40547)
I20260926 15:15:54.475970 14712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe174fe1178644bba2df4e50c0cb17f0" candidate_uuid: "5274e26d9ee742fc9d5f93465ff450e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e"
I20260926 15:15:54.476001 14851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe174fe1178644bba2df4e50c0cb17f0" candidate_uuid: "5274e26d9ee742fc9d5f93465ff450e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b7fefcc60a4f6fae92bc19687153a0"
I20260926 15:15:54.476126 14712 raft_consensus.cc:3037] T fe174fe1178644bba2df4e50c0cb17f0 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:54.476135 14851 raft_consensus.cc:3037] T fe174fe1178644bba2df4e50c0cb17f0 P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:54.477851 14712 raft_consensus.cc:2454] T fe174fe1178644bba2df4e50c0cb17f0 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5274e26d9ee742fc9d5f93465ff450e8 in term 1.
I20260926 15:15:54.477914 14851 raft_consensus.cc:2454] T fe174fe1178644bba2df4e50c0cb17f0 P b6b7fefcc60a4f6fae92bc19687153a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5274e26d9ee742fc9d5f93465ff450e8 in term 1.
I20260926 15:15:54.478307 14918 leader_election.cc:302] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5274e26d9ee742fc9d5f93465ff450e8, b6b7fefcc60a4f6fae92bc19687153a0; no voters:
I20260926 15:15:54.478567 15066 raft_consensus.cc:2781] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:54.478971 15066 raft_consensus.cc:686] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8 [term 1 LEADER]: Becoming Leader. State: Replica: 5274e26d9ee742fc9d5f93465ff450e8, State: Running, Role: LEADER
I20260926 15:15:54.479391 15066 consensus_queue.cc:237] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.482909 14550 catalog_manager.cc:5495] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5274e26d9ee742fc9d5f93465ff450e8 (127.6.209.195). New cstate: current_term: 1 leader_uuid: "5274e26d9ee742fc9d5f93465ff450e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } health_report { overall_health: HEALTHY } } }
I20260926 15:15:54.502207 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.497411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46774 (local address 127.6.209.195:36427)
0926 15:15:54.497581 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:54.497586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:54.497603 (+ 17us) server_negotiation.cc:407] Connection header received
0926 15:15:54.497809 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:54.497813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:54.497892 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:54.498041 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:15:54.498376 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.499028 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.500243 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.500608 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.500725 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.501234 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:15:54.501663 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:15:54.501815 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:54.501891 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 15:15:54.501960 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 15:15:54.509145 14851 raft_consensus.cc:1261] T fe174fe1178644bba2df4e50c0cb17f0 P b6b7fefcc60a4f6fae92bc19687153a0 [term 1 FOLLOWER]: Refusing update from remote peer 5274e26d9ee742fc9d5f93465ff450e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:54.509145 14712 raft_consensus.cc:1261] T fe174fe1178644bba2df4e50c0cb17f0 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 1 FOLLOWER]: Refusing update from remote peer 5274e26d9ee742fc9d5f93465ff450e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:54.510015 15071 consensus_queue.cc:1035] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:54.510313 15066 consensus_queue.cc:1035] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:54.520309 15074 mvcc.cc:205] Tried to move back new op lower bound from 7333624850460835840 to 7333624850350305280. Current Snapshot: MvccSnapshot[applied={T|T < 7333624850460835840}]
I20260926 15:15:54.532328 14550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:54.532569 14550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:54.535730 14550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9: Sending DeleteTablet for 3 replicas of tablet fe174fe1178644bba2df4e50c0cb17f0
I20260926 15:15:54.536609 14826 tablet_service.cc:1514] Processing DeleteTablet for tablet fe174fe1178644bba2df4e50c0cb17f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:54 UTC) from {username='oryx', principal='oryx/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:56226
I20260926 15:15:54.536720 14965 tablet_service.cc:1514] Processing DeleteTablet for tablet fe174fe1178644bba2df4e50c0cb17f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:54 UTC) from {username='oryx', principal='oryx/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:46760
I20260926 15:15:54.537222 14687 tablet_service.cc:1514] Processing DeleteTablet for tablet fe174fe1178644bba2df4e50c0cb17f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:54 UTC) from {username='oryx', principal='oryx/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:56352
I20260926 15:15:54.537341 15084 tablet_replica.cc:330] T fe174fe1178644bba2df4e50c0cb17f0 P b6b7fefcc60a4f6fae92bc19687153a0: stopping tablet replica
I20260926 15:15:54.537657 15084 raft_consensus.cc:2229] T fe174fe1178644bba2df4e50c0cb17f0 P b6b7fefcc60a4f6fae92bc19687153a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:54.538173 15084 raft_consensus.cc:2258] T fe174fe1178644bba2df4e50c0cb17f0 P b6b7fefcc60a4f6fae92bc19687153a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:54.539211 14550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:15:54.540002 14550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:15:54.540096 15084 ts_tablet_manager.cc:1905] T fe174fe1178644bba2df4e50c0cb17f0 P b6b7fefcc60a4f6fae92bc19687153a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:54.542802 15086 tablet_replica.cc:330] T fe174fe1178644bba2df4e50c0cb17f0 P 92e8f5d7dc5949468ab368fdfd2b2d2e: stopping tablet replica
I20260926 15:15:54.543179 15086 raft_consensus.cc:2229] T fe174fe1178644bba2df4e50c0cb17f0 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:54.543640 15086 raft_consensus.cc:2258] T fe174fe1178644bba2df4e50c0cb17f0 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:54.548660 14687 tablet_service.cc:1467] Processing CreateTablet for tablet e5ffc853b58b40b7a402b8d74e194b36 (DEFAULT_TABLE table=test-table [id=c980047c00754b5c9315c9d3e48c5067]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:54.549222 14687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ffc853b58b40b7a402b8d74e194b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:54.549088 14964 tablet_service.cc:1467] Processing CreateTablet for tablet e5ffc853b58b40b7a402b8d74e194b36 (DEFAULT_TABLE table=test-table [id=c980047c00754b5c9315c9d3e48c5067]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:54.549629 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ffc853b58b40b7a402b8d74e194b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:54.554102 15086 ts_tablet_manager.cc:1905] T fe174fe1178644bba2df4e50c0cb17f0 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:54.555115 15061 tablet_bootstrap.cc:492] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Bootstrap starting.
I20260926 15:15:54.555264 15060 tablet_bootstrap.cc:492] T e5ffc853b58b40b7a402b8d74e194b36 P 5274e26d9ee742fc9d5f93465ff450e8: Bootstrap starting.
I20260926 15:15:54.555964 14826 tablet_service.cc:1467] Processing CreateTablet for tablet e5ffc853b58b40b7a402b8d74e194b36 (DEFAULT_TABLE table=test-table [id=c980047c00754b5c9315c9d3e48c5067]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:54.556311 14826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ffc853b58b40b7a402b8d74e194b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:54.556387 15061 tablet_bootstrap.cc:654] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:54.556561 15060 tablet_bootstrap.cc:654] T e5ffc853b58b40b7a402b8d74e194b36 P 5274e26d9ee742fc9d5f93465ff450e8: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:54.559198 15084 ts_tablet_manager.cc:1918] T fe174fe1178644bba2df4e50c0cb17f0 P b6b7fefcc60a4f6fae92bc19687153a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:15:54.559454 15084 log.cc:1196] T fe174fe1178644bba2df4e50c0cb17f0 P b6b7fefcc60a4f6fae92bc19687153a0: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/fe174fe1178644bba2df4e50c0cb17f0
I20260926 15:15:54.560084 15084 ts_tablet_manager.cc:1939] T fe174fe1178644bba2df4e50c0cb17f0 P b6b7fefcc60a4f6fae92bc19687153a0: Deleting consensus metadata
I20260926 15:15:54.562636 14537 catalog_manager.cc:4841] TS b6b7fefcc60a4f6fae92bc19687153a0 (127.6.209.194:37611): tablet fe174fe1178644bba2df4e50c0cb17f0 (table test-table [id=f7d835c929354ce3870ef84fa3d4728b]) successfully deleted
I20260926 15:15:54.563381 15063 tablet_bootstrap.cc:492] T e5ffc853b58b40b7a402b8d74e194b36 P b6b7fefcc60a4f6fae92bc19687153a0: Bootstrap starting.
I20260926 15:15:54.564877 15086 ts_tablet_manager.cc:1918] T fe174fe1178644bba2df4e50c0cb17f0 P 92e8f5d7dc5949468ab368fdfd2b2d2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:15:54.565001 15086 log.cc:1196] T fe174fe1178644bba2df4e50c0cb17f0 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-0/wal/wals/fe174fe1178644bba2df4e50c0cb17f0
I20260926 15:15:54.565093 15063 tablet_bootstrap.cc:654] T e5ffc853b58b40b7a402b8d74e194b36 P b6b7fefcc60a4f6fae92bc19687153a0: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:54.565495 15086 ts_tablet_manager.cc:1939] T fe174fe1178644bba2df4e50c0cb17f0 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Deleting consensus metadata
I20260926 15:15:54.565531 15085 tablet_replica.cc:330] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8: stopping tablet replica
I20260926 15:15:54.565871 15085 raft_consensus.cc:2229] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:15:54.566349 15085 raft_consensus.cc:2258] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:54.566917 14535 catalog_manager.cc:4841] TS 92e8f5d7dc5949468ab368fdfd2b2d2e (127.6.209.193:40547): tablet fe174fe1178644bba2df4e50c0cb17f0 (table test-table [id=f7d835c929354ce3870ef84fa3d4728b]) successfully deleted
I20260926 15:15:54.568946 15085 ts_tablet_manager.cc:1905] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:54.569286 15061 tablet_bootstrap.cc:492] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e: No bootstrap required, opened a new log
I20260926 15:15:54.569483 15061 ts_tablet_manager.cc:1397] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.012s
I20260926 15:15:54.570415 15061 raft_consensus.cc:348] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.570732 15061 raft_consensus.cc:374] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:54.570887 15061 raft_consensus.cc:729] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e8f5d7dc5949468ab368fdfd2b2d2e, State: Initialized, Role: FOLLOWER
I20260926 15:15:54.571218 15061 consensus_queue.cc:260] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.572685 15061 ts_tablet_manager.cc:1428] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 15:15:54.573069 15060 tablet_bootstrap.cc:492] T e5ffc853b58b40b7a402b8d74e194b36 P 5274e26d9ee742fc9d5f93465ff450e8: No bootstrap required, opened a new log
I20260926 15:15:54.573202 15060 ts_tablet_manager.cc:1397] T e5ffc853b58b40b7a402b8d74e194b36 P 5274e26d9ee742fc9d5f93465ff450e8: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.000s
I20260926 15:15:54.573937 15060 raft_consensus.cc:348] T e5ffc853b58b40b7a402b8d74e194b36 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.576843 15063 tablet_bootstrap.cc:492] T e5ffc853b58b40b7a402b8d74e194b36 P b6b7fefcc60a4f6fae92bc19687153a0: No bootstrap required, opened a new log
I20260926 15:15:54.576913 15060 raft_consensus.cc:374] T e5ffc853b58b40b7a402b8d74e194b36 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:54.577023 15063 ts_tablet_manager.cc:1397] T e5ffc853b58b40b7a402b8d74e194b36 P b6b7fefcc60a4f6fae92bc19687153a0: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260926 15:15:54.577056 15060 raft_consensus.cc:729] T e5ffc853b58b40b7a402b8d74e194b36 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5274e26d9ee742fc9d5f93465ff450e8, State: Initialized, Role: FOLLOWER
I20260926 15:15:54.577376 15060 consensus_queue.cc:260] T e5ffc853b58b40b7a402b8d74e194b36 P 5274e26d9ee742fc9d5f93465ff450e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.577888 15060 ts_tablet_manager.cc:1428] T e5ffc853b58b40b7a402b8d74e194b36 P 5274e26d9ee742fc9d5f93465ff450e8: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260926 15:15:54.577792 15063 raft_consensus.cc:348] T e5ffc853b58b40b7a402b8d74e194b36 P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.578058 15063 raft_consensus.cc:374] T e5ffc853b58b40b7a402b8d74e194b36 P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:54.578192 15063 raft_consensus.cc:729] T e5ffc853b58b40b7a402b8d74e194b36 P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b7fefcc60a4f6fae92bc19687153a0, State: Initialized, Role: FOLLOWER
I20260926 15:15:54.578511 15063 consensus_queue.cc:260] T e5ffc853b58b40b7a402b8d74e194b36 P b6b7fefcc60a4f6fae92bc19687153a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.579005 15085 ts_tablet_manager.cc:1918] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:15:54.579248 15085 log.cc:1196] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/fe174fe1178644bba2df4e50c0cb17f0
I20260926 15:15:54.580598 15085 ts_tablet_manager.cc:1939] T fe174fe1178644bba2df4e50c0cb17f0 P 5274e26d9ee742fc9d5f93465ff450e8: Deleting consensus metadata
I20260926 15:15:54.581290 15063 ts_tablet_manager.cc:1428] T e5ffc853b58b40b7a402b8d74e194b36 P b6b7fefcc60a4f6fae92bc19687153a0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 15:15:54.582231 14536 catalog_manager.cc:4841] TS 5274e26d9ee742fc9d5f93465ff450e8 (127.6.209.195:36427): tablet fe174fe1178644bba2df4e50c0cb17f0 (table test-table [id=f7d835c929354ce3870ef84fa3d4728b]) successfully deleted
W20260926 15:15:54.599025 14897 tablet.cc:2369] T e5ffc853b58b40b7a402b8d74e194b36 P b6b7fefcc60a4f6fae92bc19687153a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:15:54.643071 14758 tablet.cc:2369] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:15:54.752310 15067 raft_consensus.cc:482] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:15:54.752471 15067 raft_consensus.cc:504] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.753891 15067 leader_election.cc:288] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6b7fefcc60a4f6fae92bc19687153a0 (127.6.209.194:37611), 5274e26d9ee742fc9d5f93465ff450e8 (127.6.209.195:36427)
I20260926 15:15:54.760790 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.754715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:40681 (local address 127.6.209.195:36427)
0926 15:15:54.754881 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:54.754886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:54.754903 (+ 17us) server_negotiation.cc:407] Connection header received
0926 15:15:54.755119 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:54.755128 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:54.755195 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:54.755260 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:54.755963 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.757076 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.758909 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.759844 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.759980 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.760049 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:54.760365 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 15:15:54.760486 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 15:15:54.760869 15091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.754271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:36427 (local address 127.6.209.193:40681)
0926 15:15:54.754849 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:54.754863 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:54.754945 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:54.755439 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:54.755445 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:54.755467 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:54.755810 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:54.755817 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.757251 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.757255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:54.758758 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:54.758769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.760035 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.760041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:54.760206 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.760239 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 15:15:54.760326 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 15:15:54.760449 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":57,"thread_start_us":625,"threads_started":1}
I20260926 15:15:54.761622 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.755040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:53591 (local address 127.6.209.194:37611)
0926 15:15:54.755187 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:54.755191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:54.755216 (+ 25us) server_negotiation.cc:407] Connection header received
0926 15:15:54.755275 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:54.755278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:54.755333 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:54.755408 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:54.756105 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.757075 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.759514 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.760441 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.760574 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.760638 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:54.761224 (+ 586us) server_negotiation.cc:299] Negotiation successful
0926 15:15:54.761353 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 15:15:54.761664 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.754062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:37611 (local address 127.6.209.193:53591)
0926 15:15:54.754634 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:54.754654 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:54.754778 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:54.755452 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:54.755458 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:54.755485 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:54.755957 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:54.755969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.757204 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.757211 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:54.759363 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:54.759376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.760801 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.760810 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:54.761034 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.761060 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 15:15:54.761174 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 15:15:54.761305 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":17,"thread_start_us":292,"threads_started":1}
I20260926 15:15:54.761806 14990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ffc853b58b40b7a402b8d74e194b36" candidate_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5274e26d9ee742fc9d5f93465ff450e8" is_pre_election: true
I20260926 15:15:54.762269 14990 raft_consensus.cc:2454] T e5ffc853b58b40b7a402b8d74e194b36 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92e8f5d7dc5949468ab368fdfd2b2d2e in term 0.
I20260926 15:15:54.762447 14851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ffc853b58b40b7a402b8d74e194b36" candidate_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" is_pre_election: true
I20260926 15:15:54.762672 14851 raft_consensus.cc:2454] T e5ffc853b58b40b7a402b8d74e194b36 P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92e8f5d7dc5949468ab368fdfd2b2d2e in term 0.
I20260926 15:15:54.762884 14639 leader_election.cc:302] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5274e26d9ee742fc9d5f93465ff450e8, 92e8f5d7dc5949468ab368fdfd2b2d2e; no voters:
I20260926 15:15:54.763219 15067 raft_consensus.cc:2781] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:15:54.763320 15067 raft_consensus.cc:482] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:15:54.763394 15067 raft_consensus.cc:3037] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:54.765237 15067 raft_consensus.cc:504] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.765713 15067 leader_election.cc:288] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e [CANDIDATE]: Term 1 election: Requested vote from peers b6b7fefcc60a4f6fae92bc19687153a0 (127.6.209.194:37611), 5274e26d9ee742fc9d5f93465ff450e8 (127.6.209.195:36427)
I20260926 15:15:54.766076 14851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ffc853b58b40b7a402b8d74e194b36" candidate_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b7fefcc60a4f6fae92bc19687153a0"
I20260926 15:15:54.766137 14990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ffc853b58b40b7a402b8d74e194b36" candidate_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5274e26d9ee742fc9d5f93465ff450e8"
I20260926 15:15:54.766253 14851 raft_consensus.cc:3037] T e5ffc853b58b40b7a402b8d74e194b36 P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:54.766301 14990 raft_consensus.cc:3037] T e5ffc853b58b40b7a402b8d74e194b36 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:54.767954 14851 raft_consensus.cc:2454] T e5ffc853b58b40b7a402b8d74e194b36 P b6b7fefcc60a4f6fae92bc19687153a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92e8f5d7dc5949468ab368fdfd2b2d2e in term 1.
I20260926 15:15:54.768306 14640 leader_election.cc:302] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92e8f5d7dc5949468ab368fdfd2b2d2e, b6b7fefcc60a4f6fae92bc19687153a0; no voters:
I20260926 15:15:54.768584 15067 raft_consensus.cc:2781] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:54.769088 15067 raft_consensus.cc:686] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 1 LEADER]: Becoming Leader. State: Replica: 92e8f5d7dc5949468ab368fdfd2b2d2e, State: Running, Role: LEADER
I20260926 15:15:54.769380 14990 raft_consensus.cc:2454] T e5ffc853b58b40b7a402b8d74e194b36 P 5274e26d9ee742fc9d5f93465ff450e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92e8f5d7dc5949468ab368fdfd2b2d2e in term 1.
I20260926 15:15:54.769750 15067 consensus_queue.cc:237] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.772720 14551 catalog_manager.cc:5495] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 92e8f5d7dc5949468ab368fdfd2b2d2e (127.6.209.193). New cstate: current_term: 1 leader_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } health_report { overall_health: UNKNOWN } } }
W20260926 15:15:54.812763 15036 tablet.cc:2369] T e5ffc853b58b40b7a402b8d74e194b36 P 5274e26d9ee742fc9d5f93465ff450e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 15:15:54 dist-test-slave-j6q7 krb5kdc[14506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435751, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.254@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM
I20260926 15:15:54.832471 15099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.819004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:38245 (local address 127.0.0.1:42806)
0926 15:15:54.819673 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:54.819701 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:54.819802 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:54.820172 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:54.820176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:54.820935 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:54.821172 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:54.821180 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.822452 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.822455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:54.823249 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:54.823257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.824181 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.824185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:54.824313 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.824967 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:54.824994 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:54.827056 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:54.830463 (+ 3407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:54.830471 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:54.830476 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:54.830811 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:54.831149 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:54.831152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:54.831154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:54.831259 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:54.831804 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:54.831809 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:54.831921 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 15:15:54.832037 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 15:15:54.832191 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":408,"threads_started":1}
I20260926 15:15:54.832894 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.818985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42806 (local address 127.6.209.254:38245)
0926 15:15:54.819255 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:54.819263 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:54.819774 (+ 511us) server_negotiation.cc:407] Connection header received
0926 15:15:54.819939 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:54.819945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:54.820039 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:54.820146 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:54.821303 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.822346 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.823388 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.824025 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.824150 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.827209 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:54.827216 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:54.827220 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:54.827250 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:54.830345 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:54.830918 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:54.830923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:54.830926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:54.831051 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:54.831360 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:54.831367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:54.831369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:54.831689 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:54.831829 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:54.832217 (+ 388us) server_negotiation.cc:299] Negotiation successful
0926 15:15:54.832425 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":110}
I20260926 15:15:54.837693 14551 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:42806:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 15:15:54.838682 14551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:15:54.844884 14687 tablet_service.cc:1467] Processing CreateTablet for tablet 4440b353349b41028c2ae61cf9657684 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d66167083934ce89b89b3735d660a70]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:15:54.844931 14826 tablet_service.cc:1467] Processing CreateTablet for tablet 4440b353349b41028c2ae61cf9657684 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d66167083934ce89b89b3735d660a70]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:15:54.845381 14687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4440b353349b41028c2ae61cf9657684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:54.845433 14826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4440b353349b41028c2ae61cf9657684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:54.849644 14964 tablet_service.cc:1467] Processing CreateTablet for tablet 4440b353349b41028c2ae61cf9657684 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d66167083934ce89b89b3735d660a70]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:15:54.850147 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4440b353349b41028c2ae61cf9657684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:54.861133 15061 tablet_bootstrap.cc:492] T 4440b353349b41028c2ae61cf9657684 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Bootstrap starting.
I20260926 15:15:54.862797 15061 tablet_bootstrap.cc:654] T 4440b353349b41028c2ae61cf9657684 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:54.862996 15063 tablet_bootstrap.cc:492] T 4440b353349b41028c2ae61cf9657684 P b6b7fefcc60a4f6fae92bc19687153a0: Bootstrap starting.
I20260926 15:15:54.864261 15063 tablet_bootstrap.cc:654] T 4440b353349b41028c2ae61cf9657684 P b6b7fefcc60a4f6fae92bc19687153a0: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:54.868005 15060 tablet_bootstrap.cc:492] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8: Bootstrap starting.
I20260926 15:15:54.869649 15060 tablet_bootstrap.cc:654] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:54.875464 15063 tablet_bootstrap.cc:492] T 4440b353349b41028c2ae61cf9657684 P b6b7fefcc60a4f6fae92bc19687153a0: No bootstrap required, opened a new log
I20260926 15:15:54.875607 15063 ts_tablet_manager.cc:1397] T 4440b353349b41028c2ae61cf9657684 P b6b7fefcc60a4f6fae92bc19687153a0: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20260926 15:15:54.876196 15063 raft_consensus.cc:348] T 4440b353349b41028c2ae61cf9657684 P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.876395 15063 raft_consensus.cc:374] T 4440b353349b41028c2ae61cf9657684 P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:54.876457 15063 raft_consensus.cc:729] T 4440b353349b41028c2ae61cf9657684 P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b7fefcc60a4f6fae92bc19687153a0, State: Initialized, Role: FOLLOWER
I20260926 15:15:54.876724 15063 consensus_queue.cc:260] T 4440b353349b41028c2ae61cf9657684 P b6b7fefcc60a4f6fae92bc19687153a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.877166 15068 tablet_replica.cc:439] T 4440b353349b41028c2ae61cf9657684 P b6b7fefcc60a4f6fae92bc19687153a0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } } }
I20260926 15:15:54.877452 15068 tablet_replica.cc:442] T 4440b353349b41028c2ae61cf9657684 P b6b7fefcc60a4f6fae92bc19687153a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:15:54.877897 15063 ts_tablet_manager.cc:1428] T 4440b353349b41028c2ae61cf9657684 P b6b7fefcc60a4f6fae92bc19687153a0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:15:54.878729 15060 tablet_bootstrap.cc:492] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8: No bootstrap required, opened a new log
I20260926 15:15:54.878899 15060 ts_tablet_manager.cc:1397] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20260926 15:15:54.879556 15060 raft_consensus.cc:348] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.879784 15060 raft_consensus.cc:374] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:54.879838 15060 raft_consensus.cc:729] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5274e26d9ee742fc9d5f93465ff450e8, State: Initialized, Role: FOLLOWER
I20260926 15:15:54.880019 15060 consensus_queue.cc:260] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.880465 15060 ts_tablet_manager.cc:1428] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 15:15:54.880673 15079 tablet_replica.cc:439] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } } }
I20260926 15:15:54.881093 15079 tablet_replica.cc:442] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:15:54.883312 15079 raft_consensus.cc:482] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:15:54.883484 15079 raft_consensus.cc:504] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.884141 15079 leader_election.cc:288] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92e8f5d7dc5949468ab368fdfd2b2d2e (127.6.209.193:40547), b6b7fefcc60a4f6fae92bc19687153a0 (127.6.209.194:37611)
I20260926 15:15:54.884496 14712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4440b353349b41028c2ae61cf9657684" candidate_uuid: "5274e26d9ee742fc9d5f93465ff450e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" is_pre_election: true
I20260926 15:15:54.884606 14851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4440b353349b41028c2ae61cf9657684" candidate_uuid: "5274e26d9ee742fc9d5f93465ff450e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" is_pre_election: true
I20260926 15:15:54.884872 14851 raft_consensus.cc:2454] T 4440b353349b41028c2ae61cf9657684 P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5274e26d9ee742fc9d5f93465ff450e8 in term 0.
W20260926 15:15:54.885241 14916 leader_election.cc:341] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92e8f5d7dc5949468ab368fdfd2b2d2e (127.6.209.193:40547): Illegal state: must be running to vote when last-logged opid is not known
I20260926 15:15:54.885524 14918 leader_election.cc:302] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5274e26d9ee742fc9d5f93465ff450e8, b6b7fefcc60a4f6fae92bc19687153a0; no voters: 92e8f5d7dc5949468ab368fdfd2b2d2e
I20260926 15:15:54.885914 15079 raft_consensus.cc:2781] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:15:54.886039 15079 raft_consensus.cc:482] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:15:54.886139 15079 raft_consensus.cc:3037] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:54.887151 15061 tablet_bootstrap.cc:492] T 4440b353349b41028c2ae61cf9657684 P 92e8f5d7dc5949468ab368fdfd2b2d2e: No bootstrap required, opened a new log
I20260926 15:15:54.887322 15061 ts_tablet_manager.cc:1397] T 4440b353349b41028c2ae61cf9657684 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.005s
I20260926 15:15:54.888054 15061 raft_consensus.cc:348] T 4440b353349b41028c2ae61cf9657684 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.888285 15061 raft_consensus.cc:374] T 4440b353349b41028c2ae61cf9657684 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:54.888370 15061 raft_consensus.cc:729] T 4440b353349b41028c2ae61cf9657684 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e8f5d7dc5949468ab368fdfd2b2d2e, State: Initialized, Role: FOLLOWER
I20260926 15:15:54.888496 15079 raft_consensus.cc:504] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.888654 15061 consensus_queue.cc:260] T 4440b353349b41028c2ae61cf9657684 P 92e8f5d7dc5949468ab368fdfd2b2d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.889185 15067 tablet_replica.cc:439] T 4440b353349b41028c2ae61cf9657684 P 92e8f5d7dc5949468ab368fdfd2b2d2e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } } }
I20260926 15:15:54.889345 15061 ts_tablet_manager.cc:1428] T 4440b353349b41028c2ae61cf9657684 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:15:54.889441 15067 tablet_replica.cc:442] T 4440b353349b41028c2ae61cf9657684 P 92e8f5d7dc5949468ab368fdfd2b2d2e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:15:54.889516 14851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4440b353349b41028c2ae61cf9657684" candidate_uuid: "5274e26d9ee742fc9d5f93465ff450e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b7fefcc60a4f6fae92bc19687153a0"
I20260926 15:15:54.889688 14851 raft_consensus.cc:3037] T 4440b353349b41028c2ae61cf9657684 P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:54.890349 14712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4440b353349b41028c2ae61cf9657684" candidate_uuid: "5274e26d9ee742fc9d5f93465ff450e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e"
I20260926 15:15:54.890548 14712 raft_consensus.cc:3037] T 4440b353349b41028c2ae61cf9657684 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:54.892139 14851 raft_consensus.cc:2454] T 4440b353349b41028c2ae61cf9657684 P b6b7fefcc60a4f6fae92bc19687153a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5274e26d9ee742fc9d5f93465ff450e8 in term 1.
I20260926 15:15:54.892405 15079 leader_election.cc:288] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8 [CANDIDATE]: Term 1 election: Requested vote from peers 92e8f5d7dc5949468ab368fdfd2b2d2e (127.6.209.193:40547), b6b7fefcc60a4f6fae92bc19687153a0 (127.6.209.194:37611)
I20260926 15:15:54.892620 14918 leader_election.cc:302] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5274e26d9ee742fc9d5f93465ff450e8, b6b7fefcc60a4f6fae92bc19687153a0; no voters:
I20260926 15:15:54.892935 15079 raft_consensus.cc:2781] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:54.893078 14712 raft_consensus.cc:2454] T 4440b353349b41028c2ae61cf9657684 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5274e26d9ee742fc9d5f93465ff450e8 in term 1.
I20260926 15:15:54.893256 15079 raft_consensus.cc:686] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8 [term 1 LEADER]: Becoming Leader. State: Replica: 5274e26d9ee742fc9d5f93465ff450e8, State: Running, Role: LEADER
I20260926 15:15:54.893541 15079 consensus_queue.cc:237] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:54.895092 15066 tablet_replica.cc:439] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8: TxnStatusTablet state changed. Reason: New leader 5274e26d9ee742fc9d5f93465ff450e8. Latest consensus state: current_term: 1 leader_uuid: "5274e26d9ee742fc9d5f93465ff450e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } } }
I20260926 15:15:54.895382 15066 tablet_replica.cc:442] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:15:54.896050 15104 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 15:15:54.896297 15104 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 15:15:54.896678 14551 catalog_manager.cc:5495] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5274e26d9ee742fc9d5f93465ff450e8 (127.6.209.195). New cstate: current_term: 1 leader_uuid: "5274e26d9ee742fc9d5f93465ff450e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } health_report { overall_health: HEALTHY } } }
I20260926 15:15:54.911477 15099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.906565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:36427 (local address 127.0.0.1:46786)
0926 15:15:54.906826 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:54.906839 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:54.906933 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:54.907267 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:54.907270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:54.907278 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:15:54.907403 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:54.907409 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.908776 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.908780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:54.909716 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:54.909725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.910581 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.910585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:54.910687 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.910787 (+ 100us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:15:54.911156 (+ 369us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:15:54.911167 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 15:15:54.911218 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 15:15:54.911308 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260926 15:15:54.911540 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.906548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.6.209.195:36427)
0926 15:15:54.906741 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:54.906752 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:54.906908 (+ 156us) server_negotiation.cc:407] Connection header received
0926 15:15:54.907074 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:54.907079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:54.907155 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:54.907239 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:15:54.907515 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.908664 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.909834 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.910441 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.910575 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.910952 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:15:54.911075 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:15:54.911158 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:54.911238 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 15:15:54.911311 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 15:15:54.914592 14851 raft_consensus.cc:1261] T 4440b353349b41028c2ae61cf9657684 P b6b7fefcc60a4f6fae92bc19687153a0 [term 1 FOLLOWER]: Refusing update from remote peer 5274e26d9ee742fc9d5f93465ff450e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:54.914633 14712 raft_consensus.cc:1261] T 4440b353349b41028c2ae61cf9657684 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 1 FOLLOWER]: Refusing update from remote peer 5274e26d9ee742fc9d5f93465ff450e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:54.915091 15079 consensus_queue.cc:1035] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:54.915373 15066 consensus_queue.cc:1035] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:54.919176 15071 tablet_replica.cc:439] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5274e26d9ee742fc9d5f93465ff450e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } } }
I20260926 15:15:54.919564 15071 tablet_replica.cc:442] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:15:54.919781 15076 mvcc.cc:205] Tried to move back new op lower bound from 7333624852125962240 to 7333624852046274560. Current Snapshot: MvccSnapshot[applied={T|T < 7333624852125962240}]
I20260926 15:15:54.920065 15092 tablet_replica.cc:439] T 4440b353349b41028c2ae61cf9657684 P 92e8f5d7dc5949468ab368fdfd2b2d2e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5274e26d9ee742fc9d5f93465ff450e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } } }
I20260926 15:15:54.920336 15092 tablet_replica.cc:442] T 4440b353349b41028c2ae61cf9657684 P 92e8f5d7dc5949468ab368fdfd2b2d2e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:15:54.920446 15067 tablet_replica.cc:439] T 4440b353349b41028c2ae61cf9657684 P 92e8f5d7dc5949468ab368fdfd2b2d2e: TxnStatusTablet state changed. Reason: New leader 5274e26d9ee742fc9d5f93465ff450e8. Latest consensus state: current_term: 1 leader_uuid: "5274e26d9ee742fc9d5f93465ff450e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } } }
I20260926 15:15:54.920652 15067 tablet_replica.cc:442] T 4440b353349b41028c2ae61cf9657684 P 92e8f5d7dc5949468ab368fdfd2b2d2e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:15:54.929163 15068 tablet_replica.cc:439] T 4440b353349b41028c2ae61cf9657684 P b6b7fefcc60a4f6fae92bc19687153a0: TxnStatusTablet state changed. Reason: New leader 5274e26d9ee742fc9d5f93465ff450e8. Latest consensus state: current_term: 1 leader_uuid: "5274e26d9ee742fc9d5f93465ff450e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } } }
I20260926 15:15:54.929432 15068 tablet_replica.cc:442] T 4440b353349b41028c2ae61cf9657684 P b6b7fefcc60a4f6fae92bc19687153a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:15:54.930271 15066 tablet_replica.cc:439] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5274e26d9ee742fc9d5f93465ff450e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } } }
I20260926 15:15:54.930502 15066 tablet_replica.cc:442] T 4440b353349b41028c2ae61cf9657684 P 5274e26d9ee742fc9d5f93465ff450e8: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:15:54.931474 15068 tablet_replica.cc:439] T 4440b353349b41028c2ae61cf9657684 P b6b7fefcc60a4f6fae92bc19687153a0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5274e26d9ee742fc9d5f93465ff450e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } } }
I20260926 15:15:54.931658 15068 tablet_replica.cc:442] T 4440b353349b41028c2ae61cf9657684 P b6b7fefcc60a4f6fae92bc19687153a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:15:54.935171 15058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.928964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56368 (local address 127.6.209.193:40547)
0926 15:15:54.929653 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:54.929659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:54.929678 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:15:54.929749 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:54.929753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:54.929836 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:54.929961 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:15:54.930415 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.931317 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.933011 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.933543 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.933654 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.933982 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:15:54.934604 (+ 622us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:15:54.934741 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:54.934795 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 15:15:54.934879 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590}
I20260926 15:15:54.948719 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.943915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:53985 (local address 127.6.209.195:36427)
0926 15:15:54.944061 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:54.944064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:54.944553 (+ 489us) server_negotiation.cc:407] Connection header received
0926 15:15:54.944723 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:54.944727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:54.944787 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:54.944889 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:15:54.945274 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.945881 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.947076 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.947474 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.947619 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.947994 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:15:54.948139 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:15:54.948233 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:54.948367 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 15:15:54.948434 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 15:15:54.948855 15110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.943972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:36427 (local address 127.6.209.193:53985)
0926 15:15:54.944468 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:54.944481 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:54.944613 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:54.944900 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:54.944904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:54.944914 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:15:54.945135 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:54.945141 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.946006 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.946012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:54.946956 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:54.946966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.947647 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.947653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:54.947780 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.947819 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:15:54.948245 (+ 426us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:15:54.948258 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 15:15:54.948346 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 15:15:54.948464 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":299,"threads_started":1}
I20260926 15:15:54.956715 14851 raft_consensus.cc:1261] T e5ffc853b58b40b7a402b8d74e194b36 P b6b7fefcc60a4f6fae92bc19687153a0 [term 1 FOLLOWER]: Refusing update from remote peer 92e8f5d7dc5949468ab368fdfd2b2d2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:54.957031 14990 raft_consensus.cc:1261] T e5ffc853b58b40b7a402b8d74e194b36 P 5274e26d9ee742fc9d5f93465ff450e8 [term 1 FOLLOWER]: Refusing update from remote peer 92e8f5d7dc5949468ab368fdfd2b2d2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:54.957333 15092 consensus_queue.cc:1035] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:54.957623 15067 consensus_queue.cc:1035] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:54.966672 15075 mvcc.cc:205] Tried to move back new op lower bound from 7333624852297715712 to 7333624851539574784. Current Snapshot: MvccSnapshot[applied={T|T < 7333624852297715712}]
I20260926 15:15:54.991318 15058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.985903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:42779 (local address 127.6.209.193:40547)
0926 15:15:54.986060 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:54.986068 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:54.986481 (+ 413us) server_negotiation.cc:407] Connection header received
0926 15:15:54.986717 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:54.986720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:54.986785 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:54.986861 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:15:54.987416 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.988079 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.989620 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.989990 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.990097 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.990627 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:15:54.990745 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:15:54.990826 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:54.990967 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 15:15:54.991035 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 15:15:54.991523 15115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:54.985867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:40547 (local address 127.6.209.195:42779)
0926 15:15:54.986387 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:54.986408 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:54.986530 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:54.987021 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:54.987024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:54.987032 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:15:54.987236 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:54.987246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.988247 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.988251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:54.989493 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:54.989503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:54.990130 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:54.990137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:54.990258 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:54.990338 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:15:54.990840 (+ 502us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:15:54.990850 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 15:15:54.990934 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 15:15:54.991047 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":218,"threads_started":1}
I20260926 15:15:55.022379 14551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:55.022594 14551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:55.025457 14551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9: Sending DeleteTablet for 3 replicas of tablet e5ffc853b58b40b7a402b8d74e194b36
I20260926 15:15:55.026021 14687 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ffc853b58b40b7a402b8d74e194b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:55 UTC) from {username='oryx', principal='oryx/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:56352
I20260926 15:15:55.026244 15086 tablet_replica.cc:330] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e: stopping tablet replica
I20260926 15:15:55.026366 14826 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ffc853b58b40b7a402b8d74e194b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:55 UTC) from {username='oryx', principal='oryx/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:56226
I20260926 15:15:55.026407 15086 raft_consensus.cc:2229] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:15:55.026610 15084 tablet_replica.cc:330] T e5ffc853b58b40b7a402b8d74e194b36 P b6b7fefcc60a4f6fae92bc19687153a0: stopping tablet replica
I20260926 15:15:55.026769 15084 raft_consensus.cc:2229] T e5ffc853b58b40b7a402b8d74e194b36 P b6b7fefcc60a4f6fae92bc19687153a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:55.026842 15086 raft_consensus.cc:2258] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:55.026916 15084 raft_consensus.cc:2258] T e5ffc853b58b40b7a402b8d74e194b36 P b6b7fefcc60a4f6fae92bc19687153a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:55.028151 15084 ts_tablet_manager.cc:1905] T e5ffc853b58b40b7a402b8d74e194b36 P b6b7fefcc60a4f6fae92bc19687153a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:55.028965 15086 ts_tablet_manager.cc:1905] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:55.037230 14964 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ffc853b58b40b7a402b8d74e194b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:55 UTC) from {username='oryx', principal='oryx/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:46760
I20260926 15:15:55.037478 15085 tablet_replica.cc:330] T e5ffc853b58b40b7a402b8d74e194b36 P 5274e26d9ee742fc9d5f93465ff450e8: stopping tablet replica
I20260926 15:15:55.037657 15085 raft_consensus.cc:2229] T e5ffc853b58b40b7a402b8d74e194b36 P 5274e26d9ee742fc9d5f93465ff450e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:55.037830 15085 raft_consensus.cc:2258] T e5ffc853b58b40b7a402b8d74e194b36 P 5274e26d9ee742fc9d5f93465ff450e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:55.039098 15085 ts_tablet_manager.cc:1905] T e5ffc853b58b40b7a402b8d74e194b36 P 5274e26d9ee742fc9d5f93465ff450e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:55.045282 15086 ts_tablet_manager.cc:1918] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:15:55.045411 15086 log.cc:1196] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-0/wal/wals/e5ffc853b58b40b7a402b8d74e194b36
I20260926 15:15:55.045805 15086 ts_tablet_manager.cc:1939] T e5ffc853b58b40b7a402b8d74e194b36 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Deleting consensus metadata
I20260926 15:15:55.046734 14535 catalog_manager.cc:4841] TS 92e8f5d7dc5949468ab368fdfd2b2d2e (127.6.209.193:40547): tablet e5ffc853b58b40b7a402b8d74e194b36 (table test-table [id=c980047c00754b5c9315c9d3e48c5067]) successfully deleted
I20260926 15:15:55.047272 15084 ts_tablet_manager.cc:1918] T e5ffc853b58b40b7a402b8d74e194b36 P b6b7fefcc60a4f6fae92bc19687153a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:15:55.047421 15085 ts_tablet_manager.cc:1918] T e5ffc853b58b40b7a402b8d74e194b36 P 5274e26d9ee742fc9d5f93465ff450e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:15:55.047592 15084 log.cc:1196] T e5ffc853b58b40b7a402b8d74e194b36 P b6b7fefcc60a4f6fae92bc19687153a0: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/e5ffc853b58b40b7a402b8d74e194b36
I20260926 15:15:55.047746 15085 log.cc:1196] T e5ffc853b58b40b7a402b8d74e194b36 P 5274e26d9ee742fc9d5f93465ff450e8: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/e5ffc853b58b40b7a402b8d74e194b36
I20260926 15:15:55.048169 15084 ts_tablet_manager.cc:1939] T e5ffc853b58b40b7a402b8d74e194b36 P b6b7fefcc60a4f6fae92bc19687153a0: Deleting consensus metadata
I20260926 15:15:55.048601 15085 ts_tablet_manager.cc:1939] T e5ffc853b58b40b7a402b8d74e194b36 P 5274e26d9ee742fc9d5f93465ff450e8: Deleting consensus metadata
I20260926 15:15:55.049683 14537 catalog_manager.cc:4841] TS b6b7fefcc60a4f6fae92bc19687153a0 (127.6.209.194:37611): tablet e5ffc853b58b40b7a402b8d74e194b36 (table test-table [id=c980047c00754b5c9315c9d3e48c5067]) successfully deleted
I20260926 15:15:55.050621 14536 catalog_manager.cc:4841] TS 5274e26d9ee742fc9d5f93465ff450e8 (127.6.209.195:36427): tablet e5ffc853b58b40b7a402b8d74e194b36 (table test-table [id=c980047c00754b5c9315c9d3e48c5067]) successfully deleted
Sep 26 15:15:55 dist-test-slave-j6q7 krb5kdc[14506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 15:15:55 dist-test-slave-j6q7 krb5kdc[14506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:15:55 dist-test-slave-j6q7 krb5kdc[14506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 15:15:55.095125 14599 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:15:55.083192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42808 (local address 127.6.209.254:38245)
0926 15:15:55.083338 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:55.083344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:55.083785 (+ 441us) server_negotiation.cc:407] Connection header received
0926 15:15:55.083970 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:55.083975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:55.084052 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:55.084150 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:55.085714 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:55.086762 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:55.088482 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:55.089163 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:55.089284 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:55.094890 (+ 5606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42808: BlockingRecv error: recv got EOF from 127.0.0.1:42808 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 15:15:55 dist-test-slave-j6q7 krb5kdc[14506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM
I20260926 15:15:55.114262 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:55.097786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42818 (local address 127.6.209.254:38245)
0926 15:15:55.097919 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:55.097924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:55.098650 (+ 726us) server_negotiation.cc:407] Connection header received
0926 15:15:55.098781 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:55.098787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:55.098854 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:55.098950 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:55.100292 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:55.101407 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:55.103250 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:55.103867 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:55.103987 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:55.108243 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:55.108249 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:55.108254 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:55.108292 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:55.111617 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:55.112296 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:55.112301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:55.112305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:55.112441 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:55.112859 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:55.112864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:55.112867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:55.113160 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:55.113378 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:55.113713 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 15:15:55.113923 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 15:15:55.117762 14551 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:15:55.118445 14551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:15:55.125490 14826 tablet_service.cc:1467] Processing CreateTablet for tablet d4354b75074240a4aff4289de9214a01 (DEFAULT_TABLE table=test-table [id=1d559811eaa348d6977435282a898755]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:55.125761 14964 tablet_service.cc:1467] Processing CreateTablet for tablet d4354b75074240a4aff4289de9214a01 (DEFAULT_TABLE table=test-table [id=1d559811eaa348d6977435282a898755]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:55.125882 14826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4354b75074240a4aff4289de9214a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:55.126142 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4354b75074240a4aff4289de9214a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:55.132571 15063 tablet_bootstrap.cc:492] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0: Bootstrap starting.
I20260926 15:15:55.132519 14687 tablet_service.cc:1467] Processing CreateTablet for tablet d4354b75074240a4aff4289de9214a01 (DEFAULT_TABLE table=test-table [id=1d559811eaa348d6977435282a898755]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:55.132921 14687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4354b75074240a4aff4289de9214a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:55.134102 15063 tablet_bootstrap.cc:654] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:55.141448 15060 tablet_bootstrap.cc:492] T d4354b75074240a4aff4289de9214a01 P 5274e26d9ee742fc9d5f93465ff450e8: Bootstrap starting.
I20260926 15:15:55.141644 15061 tablet_bootstrap.cc:492] T d4354b75074240a4aff4289de9214a01 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Bootstrap starting.
I20260926 15:15:55.142952 15060 tablet_bootstrap.cc:654] T d4354b75074240a4aff4289de9214a01 P 5274e26d9ee742fc9d5f93465ff450e8: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:55.143092 15061 tablet_bootstrap.cc:654] T d4354b75074240a4aff4289de9214a01 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:55.148096 15063 tablet_bootstrap.cc:492] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0: No bootstrap required, opened a new log
I20260926 15:15:55.148559 15063 ts_tablet_manager.cc:1397] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.002s
I20260926 15:15:55.149503 15063 raft_consensus.cc:348] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:55.150751 15063 raft_consensus.cc:374] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:55.150977 15063 raft_consensus.cc:729] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b7fefcc60a4f6fae92bc19687153a0, State: Initialized, Role: FOLLOWER
I20260926 15:15:55.151378 15063 consensus_queue.cc:260] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:55.154234 15063 ts_tablet_manager.cc:1428] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260926 15:15:55.159300 15060 tablet_bootstrap.cc:492] T d4354b75074240a4aff4289de9214a01 P 5274e26d9ee742fc9d5f93465ff450e8: No bootstrap required, opened a new log
I20260926 15:15:55.159497 15060 ts_tablet_manager.cc:1397] T d4354b75074240a4aff4289de9214a01 P 5274e26d9ee742fc9d5f93465ff450e8: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.009s
I20260926 15:15:55.160259 15060 raft_consensus.cc:348] T d4354b75074240a4aff4289de9214a01 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:55.160564 15060 raft_consensus.cc:374] T d4354b75074240a4aff4289de9214a01 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:55.160687 15060 raft_consensus.cc:729] T d4354b75074240a4aff4289de9214a01 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5274e26d9ee742fc9d5f93465ff450e8, State: Initialized, Role: FOLLOWER
I20260926 15:15:55.161059 15060 consensus_queue.cc:260] T d4354b75074240a4aff4289de9214a01 P 5274e26d9ee742fc9d5f93465ff450e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:55.163854 15060 ts_tablet_manager.cc:1428] T d4354b75074240a4aff4289de9214a01 P 5274e26d9ee742fc9d5f93465ff450e8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260926 15:15:55.166806 15061 tablet_bootstrap.cc:492] T d4354b75074240a4aff4289de9214a01 P 92e8f5d7dc5949468ab368fdfd2b2d2e: No bootstrap required, opened a new log
I20260926 15:15:55.166929 15061 ts_tablet_manager.cc:1397] T d4354b75074240a4aff4289de9214a01 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.013s
I20260926 15:15:55.167574 15061 raft_consensus.cc:348] T d4354b75074240a4aff4289de9214a01 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:55.167809 15061 raft_consensus.cc:374] T d4354b75074240a4aff4289de9214a01 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:55.167896 15061 raft_consensus.cc:729] T d4354b75074240a4aff4289de9214a01 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e8f5d7dc5949468ab368fdfd2b2d2e, State: Initialized, Role: FOLLOWER
I20260926 15:15:55.168130 15061 consensus_queue.cc:260] T d4354b75074240a4aff4289de9214a01 P 92e8f5d7dc5949468ab368fdfd2b2d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:55.168671 15061 ts_tablet_manager.cc:1428] T d4354b75074240a4aff4289de9214a01 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 15:15:55.361176 15068 raft_consensus.cc:482] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:15:55.361387 15068 raft_consensus.cc:504] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:55.362756 15068 leader_election.cc:288] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92e8f5d7dc5949468ab368fdfd2b2d2e (127.6.209.193:40547), 5274e26d9ee742fc9d5f93465ff450e8 (127.6.209.195:36427)
I20260926 15:15:55.369481 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:55.363106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:56919 (local address 127.6.209.195:36427)
0926 15:15:55.363298 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:55.363307 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:55.364179 (+ 872us) server_negotiation.cc:407] Connection header received
0926 15:15:55.364446 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:55.364453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:55.364517 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:55.364607 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:55.365232 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:55.365983 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:55.367480 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:55.368536 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:55.368692 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:55.368764 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:55.369109 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 15:15:55.369250 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:15:55.369567 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:55.363044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:36427 (local address 127.6.209.194:56919)
0926 15:15:55.364086 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:55.364105 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:55.364278 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:55.364642 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:55.364646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:55.364670 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:55.365063 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:55.365076 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:55.366095 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:55.366102 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:55.367353 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:55.367363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:55.368716 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:55.368724 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:55.368926 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:55.368964 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 15:15:55.369065 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 15:15:55.369204 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"thread_start_us":763,"threads_started":1}
I20260926 15:15:55.370319 15058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:55.363295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:36977 (local address 127.6.209.193:40547)
0926 15:15:55.363510 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:55.363515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:55.363745 (+ 230us) server_negotiation.cc:407] Connection header received
0926 15:15:55.364034 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:55.364037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:55.364095 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:55.364183 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:15:55.364687 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:55.365618 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:55.367768 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:55.369163 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:55.369342 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:55.369402 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:55.369935 (+ 533us) server_negotiation.cc:299] Negotiation successful
0926 15:15:55.370070 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 15:15:55.370328 14989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4354b75074240a4aff4289de9214a01" candidate_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5274e26d9ee742fc9d5f93465ff450e8" is_pre_election: true
I20260926 15:15:55.370570 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:55.362960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:40547 (local address 127.6.209.194:36977)
0926 15:15:55.363644 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:55.363663 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:55.363857 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:55.364198 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:55.364202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:55.364224 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:15:55.364505 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:55.364516 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:55.365765 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:55.365771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:55.367618 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:55.367631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:55.369545 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:55.369553 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:55.369715 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:55.369744 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 15:15:55.369875 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 15:15:55.370275 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":598,"threads_started":1}
I20260926 15:15:55.370589 14989 raft_consensus.cc:2454] T d4354b75074240a4aff4289de9214a01 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6b7fefcc60a4f6fae92bc19687153a0 in term 0.
I20260926 15:15:55.371089 14778 leader_election.cc:302] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5274e26d9ee742fc9d5f93465ff450e8, b6b7fefcc60a4f6fae92bc19687153a0; no voters:
I20260926 15:15:55.371150 14712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4354b75074240a4aff4289de9214a01" candidate_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" is_pre_election: true
I20260926 15:15:55.371443 15068 raft_consensus.cc:2781] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:15:55.371420 14712 raft_consensus.cc:2454] T d4354b75074240a4aff4289de9214a01 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6b7fefcc60a4f6fae92bc19687153a0 in term 0.
I20260926 15:15:55.371570 15068 raft_consensus.cc:482] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:15:55.371670 15068 raft_consensus.cc:3037] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:55.374070 15068 raft_consensus.cc:504] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:55.374501 15068 leader_election.cc:288] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0 [CANDIDATE]: Term 1 election: Requested vote from peers 92e8f5d7dc5949468ab368fdfd2b2d2e (127.6.209.193:40547), 5274e26d9ee742fc9d5f93465ff450e8 (127.6.209.195:36427)
I20260926 15:15:55.374902 14989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4354b75074240a4aff4289de9214a01" candidate_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5274e26d9ee742fc9d5f93465ff450e8"
I20260926 15:15:55.374907 14712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4354b75074240a4aff4289de9214a01" candidate_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e"
I20260926 15:15:55.375067 14989 raft_consensus.cc:3037] T d4354b75074240a4aff4289de9214a01 P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:55.375092 14712 raft_consensus.cc:3037] T d4354b75074240a4aff4289de9214a01 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:55.376957 14989 raft_consensus.cc:2454] T d4354b75074240a4aff4289de9214a01 P 5274e26d9ee742fc9d5f93465ff450e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6b7fefcc60a4f6fae92bc19687153a0 in term 1.
I20260926 15:15:55.377279 14778 leader_election.cc:302] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5274e26d9ee742fc9d5f93465ff450e8, b6b7fefcc60a4f6fae92bc19687153a0; no voters:
I20260926 15:15:55.377604 15068 raft_consensus.cc:2781] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:55.378068 15068 raft_consensus.cc:686] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0 [term 1 LEADER]: Becoming Leader. State: Replica: b6b7fefcc60a4f6fae92bc19687153a0, State: Running, Role: LEADER
I20260926 15:15:55.378088 14712 raft_consensus.cc:2454] T d4354b75074240a4aff4289de9214a01 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6b7fefcc60a4f6fae92bc19687153a0 in term 1.
I20260926 15:15:55.378464 15068 consensus_queue.cc:237] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:55.381844 14551 catalog_manager.cc:5495] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6b7fefcc60a4f6fae92bc19687153a0 (127.6.209.194). New cstate: current_term: 1 leader_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } health_report { overall_health: UNKNOWN } } }
I20260926 15:15:55.440364 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:55.435110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56238 (local address 127.6.209.194:37611)
0926 15:15:55.435631 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:55.435634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:55.435650 (+ 16us) server_negotiation.cc:407] Connection header received
0926 15:15:55.435741 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:55.435744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:55.435825 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:55.435972 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:15:55.436340 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:55.437095 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:55.438323 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:55.438728 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:55.438852 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:55.439316 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:15:55.439798 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:15:55.439931 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:55.439987 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 15:15:55.440051 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":292,"threads_started":1}
I20260926 15:15:55.443673 14989 raft_consensus.cc:1261] T d4354b75074240a4aff4289de9214a01 P 5274e26d9ee742fc9d5f93465ff450e8 [term 1 FOLLOWER]: Refusing update from remote peer b6b7fefcc60a4f6fae92bc19687153a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:55.443847 14712 raft_consensus.cc:1261] T d4354b75074240a4aff4289de9214a01 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 1 FOLLOWER]: Refusing update from remote peer b6b7fefcc60a4f6fae92bc19687153a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:55.444366 15068 consensus_queue.cc:1035] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:55.444898 15068 consensus_queue.cc:1035] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:55.463702 14550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:55.463908 14550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:55.466415 14550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9: Sending DeleteTablet for 3 replicas of tablet d4354b75074240a4aff4289de9214a01
I20260926 15:15:55.466985 14687 tablet_service.cc:1514] Processing DeleteTablet for tablet d4354b75074240a4aff4289de9214a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:55 UTC) from {username='oryx', principal='oryx/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:56352
I20260926 15:15:55.467056 14826 tablet_service.cc:1514] Processing DeleteTablet for tablet d4354b75074240a4aff4289de9214a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:55 UTC) from {username='oryx', principal='oryx/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:56226
I20260926 15:15:55.467072 14964 tablet_service.cc:1514] Processing DeleteTablet for tablet d4354b75074240a4aff4289de9214a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:55 UTC) from {username='oryx', principal='oryx/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:46760
I20260926 15:15:55.467260 15086 tablet_replica.cc:330] T d4354b75074240a4aff4289de9214a01 P 92e8f5d7dc5949468ab368fdfd2b2d2e: stopping tablet replica
I20260926 15:15:55.467312 15084 tablet_replica.cc:330] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0: stopping tablet replica
I20260926 15:15:55.467417 15086 raft_consensus.cc:2229] T d4354b75074240a4aff4289de9214a01 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:55.467450 15084 raft_consensus.cc:2229] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:15:55.467603 15086 raft_consensus.cc:2258] T d4354b75074240a4aff4289de9214a01 P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:55.467825 15084 raft_consensus.cc:2258] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:55.468089 15085 tablet_replica.cc:330] T d4354b75074240a4aff4289de9214a01 P 5274e26d9ee742fc9d5f93465ff450e8: stopping tablet replica
I20260926 15:15:55.468030 14550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 15:15:55.468242 15085 raft_consensus.cc:2229] T d4354b75074240a4aff4289de9214a01 P 5274e26d9ee742fc9d5f93465ff450e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:55.468433 15085 raft_consensus.cc:2258] T d4354b75074240a4aff4289de9214a01 P 5274e26d9ee742fc9d5f93465ff450e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:55.468497 15086 ts_tablet_manager.cc:1905] T d4354b75074240a4aff4289de9214a01 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 15:15:55.468596 14550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:15:55.469676 15085 ts_tablet_manager.cc:1905] T d4354b75074240a4aff4289de9214a01 P 5274e26d9ee742fc9d5f93465ff450e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:55.474028 15086 ts_tablet_manager.cc:1918] T d4354b75074240a4aff4289de9214a01 P 92e8f5d7dc5949468ab368fdfd2b2d2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:15:55.474205 15086 log.cc:1196] T d4354b75074240a4aff4289de9214a01 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-0/wal/wals/d4354b75074240a4aff4289de9214a01
I20260926 15:15:55.474562 15084 ts_tablet_manager.cc:1905] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:55.474591 14687 tablet_service.cc:1467] Processing CreateTablet for tablet bf2a73282df74042a6885e7b2a93bb8f (DEFAULT_TABLE table=test-table [id=9d7398d17d484751a3b5a290b2b91b99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:55.474895 15086 ts_tablet_manager.cc:1939] T d4354b75074240a4aff4289de9214a01 P 92e8f5d7dc5949468ab368fdfd2b2d2e: Deleting consensus metadata
I20260926 15:15:55.475015 14687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf2a73282df74042a6885e7b2a93bb8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:55.474985 14826 tablet_service.cc:1467] Processing CreateTablet for tablet bf2a73282df74042a6885e7b2a93bb8f (DEFAULT_TABLE table=test-table [id=9d7398d17d484751a3b5a290b2b91b99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:55.475323 14826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf2a73282df74042a6885e7b2a93bb8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:55.477396 14964 tablet_service.cc:1467] Processing CreateTablet for tablet bf2a73282df74042a6885e7b2a93bb8f (DEFAULT_TABLE table=test-table [id=9d7398d17d484751a3b5a290b2b91b99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:15:55.477735 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf2a73282df74042a6885e7b2a93bb8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:55.478965 15061 tablet_bootstrap.cc:492] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e: Bootstrap starting.
I20260926 15:15:55.479161 14535 catalog_manager.cc:4841] TS 92e8f5d7dc5949468ab368fdfd2b2d2e (127.6.209.193:40547): tablet d4354b75074240a4aff4289de9214a01 (table test-table [id=1d559811eaa348d6977435282a898755]) successfully deleted
I20260926 15:15:55.480114 15061 tablet_bootstrap.cc:654] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:55.480594 15084 ts_tablet_manager.cc:1918] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:15:55.480705 15084 log.cc:1196] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/d4354b75074240a4aff4289de9214a01
I20260926 15:15:55.481874 15061 tablet_bootstrap.cc:492] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e: No bootstrap required, opened a new log
I20260926 15:15:55.482077 15061 ts_tablet_manager.cc:1397] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 15:15:55.482556 15085 ts_tablet_manager.cc:1918] T d4354b75074240a4aff4289de9214a01 P 5274e26d9ee742fc9d5f93465ff450e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:15:55.482736 15085 log.cc:1196] T d4354b75074240a4aff4289de9214a01 P 5274e26d9ee742fc9d5f93465ff450e8: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/d4354b75074240a4aff4289de9214a01
I20260926 15:15:55.482805 15061 raft_consensus.cc:348] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:55.483013 15061 raft_consensus.cc:374] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:55.483106 15061 raft_consensus.cc:729] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e8f5d7dc5949468ab368fdfd2b2d2e, State: Initialized, Role: FOLLOWER
I20260926 15:15:55.483155 15085 ts_tablet_manager.cc:1939] T d4354b75074240a4aff4289de9214a01 P 5274e26d9ee742fc9d5f93465ff450e8: Deleting consensus metadata
I20260926 15:15:55.483337 15061 consensus_queue.cc:260] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:55.484050 15061 ts_tablet_manager.cc:1428] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:15:55.484093 14536 catalog_manager.cc:4841] TS 5274e26d9ee742fc9d5f93465ff450e8 (127.6.209.195:36427): tablet d4354b75074240a4aff4289de9214a01 (table test-table [id=1d559811eaa348d6977435282a898755]) successfully deleted
I20260926 15:15:55.484524 15084 ts_tablet_manager.cc:1939] T d4354b75074240a4aff4289de9214a01 P b6b7fefcc60a4f6fae92bc19687153a0: Deleting consensus metadata
I20260926 15:15:55.485459 14537 catalog_manager.cc:4841] TS b6b7fefcc60a4f6fae92bc19687153a0 (127.6.209.194:37611): tablet d4354b75074240a4aff4289de9214a01 (table test-table [id=1d559811eaa348d6977435282a898755]) successfully deleted
I20260926 15:15:55.488242 15063 tablet_bootstrap.cc:492] T bf2a73282df74042a6885e7b2a93bb8f P b6b7fefcc60a4f6fae92bc19687153a0: Bootstrap starting.
I20260926 15:15:55.488749 15060 tablet_bootstrap.cc:492] T bf2a73282df74042a6885e7b2a93bb8f P 5274e26d9ee742fc9d5f93465ff450e8: Bootstrap starting.
I20260926 15:15:55.489931 15063 tablet_bootstrap.cc:654] T bf2a73282df74042a6885e7b2a93bb8f P b6b7fefcc60a4f6fae92bc19687153a0: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:55.490129 15060 tablet_bootstrap.cc:654] T bf2a73282df74042a6885e7b2a93bb8f P 5274e26d9ee742fc9d5f93465ff450e8: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:55.492976 15060 tablet_bootstrap.cc:492] T bf2a73282df74042a6885e7b2a93bb8f P 5274e26d9ee742fc9d5f93465ff450e8: No bootstrap required, opened a new log
I20260926 15:15:55.493119 15060 ts_tablet_manager.cc:1397] T bf2a73282df74042a6885e7b2a93bb8f P 5274e26d9ee742fc9d5f93465ff450e8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 15:15:55.493723 15060 raft_consensus.cc:348] T bf2a73282df74042a6885e7b2a93bb8f P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:55.493918 15060 raft_consensus.cc:374] T bf2a73282df74042a6885e7b2a93bb8f P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:55.493978 15060 raft_consensus.cc:729] T bf2a73282df74042a6885e7b2a93bb8f P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5274e26d9ee742fc9d5f93465ff450e8, State: Initialized, Role: FOLLOWER
I20260926 15:15:55.494174 15060 consensus_queue.cc:260] T bf2a73282df74042a6885e7b2a93bb8f P 5274e26d9ee742fc9d5f93465ff450e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:55.495020 15060 ts_tablet_manager.cc:1428] T bf2a73282df74042a6885e7b2a93bb8f P 5274e26d9ee742fc9d5f93465ff450e8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:15:55.500140 15063 tablet_bootstrap.cc:492] T bf2a73282df74042a6885e7b2a93bb8f P b6b7fefcc60a4f6fae92bc19687153a0: No bootstrap required, opened a new log
I20260926 15:15:55.500278 15063 ts_tablet_manager.cc:1397] T bf2a73282df74042a6885e7b2a93bb8f P b6b7fefcc60a4f6fae92bc19687153a0: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260926 15:15:55.501116 15063 raft_consensus.cc:348] T bf2a73282df74042a6885e7b2a93bb8f P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:55.501396 15063 raft_consensus.cc:374] T bf2a73282df74042a6885e7b2a93bb8f P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:55.501487 15063 raft_consensus.cc:729] T bf2a73282df74042a6885e7b2a93bb8f P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b7fefcc60a4f6fae92bc19687153a0, State: Initialized, Role: FOLLOWER
I20260926 15:15:55.501729 15063 consensus_queue.cc:260] T bf2a73282df74042a6885e7b2a93bb8f P b6b7fefcc60a4f6fae92bc19687153a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:55.502171 15063 ts_tablet_manager.cc:1428] T bf2a73282df74042a6885e7b2a93bb8f P b6b7fefcc60a4f6fae92bc19687153a0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:15:55.563206 15113 raft_consensus.cc:482] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:15:55.563369 15113 raft_consensus.cc:504] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:55.563954 15113 leader_election.cc:288] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6b7fefcc60a4f6fae92bc19687153a0 (127.6.209.194:37611), 5274e26d9ee742fc9d5f93465ff450e8 (127.6.209.195:36427)
I20260926 15:15:55.564471 14989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf2a73282df74042a6885e7b2a93bb8f" candidate_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5274e26d9ee742fc9d5f93465ff450e8" is_pre_election: true
I20260926 15:15:55.564448 14851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf2a73282df74042a6885e7b2a93bb8f" candidate_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" is_pre_election: true
I20260926 15:15:55.564667 14989 raft_consensus.cc:2454] T bf2a73282df74042a6885e7b2a93bb8f P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92e8f5d7dc5949468ab368fdfd2b2d2e in term 0.
I20260926 15:15:55.564710 14851 raft_consensus.cc:2454] T bf2a73282df74042a6885e7b2a93bb8f P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92e8f5d7dc5949468ab368fdfd2b2d2e in term 0.
I20260926 15:15:55.565019 14639 leader_election.cc:302] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5274e26d9ee742fc9d5f93465ff450e8, 92e8f5d7dc5949468ab368fdfd2b2d2e; no voters:
I20260926 15:15:55.565295 15113 raft_consensus.cc:2781] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:15:55.565394 15113 raft_consensus.cc:482] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:15:55.565475 15113 raft_consensus.cc:3037] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:55.567167 15113 raft_consensus.cc:504] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:55.567596 15113 leader_election.cc:288] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e [CANDIDATE]: Term 1 election: Requested vote from peers b6b7fefcc60a4f6fae92bc19687153a0 (127.6.209.194:37611), 5274e26d9ee742fc9d5f93465ff450e8 (127.6.209.195:36427)
I20260926 15:15:55.567984 14851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf2a73282df74042a6885e7b2a93bb8f" candidate_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b7fefcc60a4f6fae92bc19687153a0"
I20260926 15:15:55.568011 14989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf2a73282df74042a6885e7b2a93bb8f" candidate_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5274e26d9ee742fc9d5f93465ff450e8"
I20260926 15:15:55.568168 14851 raft_consensus.cc:3037] T bf2a73282df74042a6885e7b2a93bb8f P b6b7fefcc60a4f6fae92bc19687153a0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:55.568220 14989 raft_consensus.cc:3037] T bf2a73282df74042a6885e7b2a93bb8f P 5274e26d9ee742fc9d5f93465ff450e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:55.570501 14851 raft_consensus.cc:2454] T bf2a73282df74042a6885e7b2a93bb8f P b6b7fefcc60a4f6fae92bc19687153a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92e8f5d7dc5949468ab368fdfd2b2d2e in term 1.
I20260926 15:15:55.570926 14989 raft_consensus.cc:2454] T bf2a73282df74042a6885e7b2a93bb8f P 5274e26d9ee742fc9d5f93465ff450e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92e8f5d7dc5949468ab368fdfd2b2d2e in term 1.
I20260926 15:15:55.571302 14639 leader_election.cc:302] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5274e26d9ee742fc9d5f93465ff450e8, 92e8f5d7dc5949468ab368fdfd2b2d2e; no voters:
I20260926 15:15:55.571545 15113 raft_consensus.cc:2781] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:55.571995 15113 raft_consensus.cc:686] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 1 LEADER]: Becoming Leader. State: Replica: 92e8f5d7dc5949468ab368fdfd2b2d2e, State: Running, Role: LEADER
I20260926 15:15:55.572237 15113 consensus_queue.cc:237] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } }
I20260926 15:15:55.574604 14550 catalog_manager.cc:5495] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 92e8f5d7dc5949468ab368fdfd2b2d2e (127.6.209.193). New cstate: current_term: 1 leader_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e8f5d7dc5949468ab368fdfd2b2d2e" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 40547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 } health_report { overall_health: UNKNOWN } } }
I20260926 15:15:55.603772 15058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:55.598665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56382 (local address 127.6.209.193:40547)
0926 15:15:55.598835 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:55.598842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:55.598930 (+ 88us) server_negotiation.cc:407] Connection header received
0926 15:15:55.599111 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:55.599117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:55.599192 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:55.599293 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:15:55.599565 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:55.600521 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:55.601986 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:55.602575 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:55.602690 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:55.603127 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:15:55.603241 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:15:55.603318 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:55.603443 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 15:15:55.603530 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 15:15:55.610687 14989 raft_consensus.cc:1261] T bf2a73282df74042a6885e7b2a93bb8f P 5274e26d9ee742fc9d5f93465ff450e8 [term 1 FOLLOWER]: Refusing update from remote peer 92e8f5d7dc5949468ab368fdfd2b2d2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:55.610975 14850 raft_consensus.cc:1261] T bf2a73282df74042a6885e7b2a93bb8f P b6b7fefcc60a4f6fae92bc19687153a0 [term 1 FOLLOWER]: Refusing update from remote peer 92e8f5d7dc5949468ab368fdfd2b2d2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:15:55.611261 15113 consensus_queue.cc:1035] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5274e26d9ee742fc9d5f93465ff450e8" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:55.611647 15113 consensus_queue.cc:1035] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b7fefcc60a4f6fae92bc19687153a0" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 37611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:15:55.667325 14550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:55.667541 14550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:15:55.670078 14550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d00a0cbd2c3049aab7d7e3d29d5568e9: Sending DeleteTablet for 3 replicas of tablet bf2a73282df74042a6885e7b2a93bb8f
I20260926 15:15:55.670781 14687 tablet_service.cc:1514] Processing DeleteTablet for tablet bf2a73282df74042a6885e7b2a93bb8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:55 UTC) from {username='oryx', principal='oryx/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:56352
I20260926 15:15:55.670818 14964 tablet_service.cc:1514] Processing DeleteTablet for tablet bf2a73282df74042a6885e7b2a93bb8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:55 UTC) from {username='oryx', principal='oryx/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:46760
I20260926 15:15:55.671016 15086 tablet_replica.cc:330] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e: stopping tablet replica
I20260926 15:15:55.671025 15085 tablet_replica.cc:330] T bf2a73282df74042a6885e7b2a93bb8f P 5274e26d9ee742fc9d5f93465ff450e8: stopping tablet replica
I20260926 15:15:55.671182 15086 raft_consensus.cc:2229] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:15:55.671195 15085 raft_consensus.cc:2229] T bf2a73282df74042a6885e7b2a93bb8f P 5274e26d9ee742fc9d5f93465ff450e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:55.671325 15085 raft_consensus.cc:2258] T bf2a73282df74042a6885e7b2a93bb8f P 5274e26d9ee742fc9d5f93465ff450e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:55.671440 15086 raft_consensus.cc:2258] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:55.671839 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 14622
I20260926 15:15:55.672225 15085 ts_tablet_manager.cc:1905] T bf2a73282df74042a6885e7b2a93bb8f P 5274e26d9ee742fc9d5f93465ff450e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:55.672325 15086 ts_tablet_manager.cc:1905] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:55.674019 14826 tablet_service.cc:1514] Processing DeleteTablet for tablet bf2a73282df74042a6885e7b2a93bb8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:15:55 UTC) from {username='oryx', principal='oryx/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:56226
I20260926 15:15:55.675072 15084 tablet_replica.cc:330] T bf2a73282df74042a6885e7b2a93bb8f P b6b7fefcc60a4f6fae92bc19687153a0: stopping tablet replica
I20260926 15:15:55.675249 15084 raft_consensus.cc:2229] T bf2a73282df74042a6885e7b2a93bb8f P b6b7fefcc60a4f6fae92bc19687153a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:15:55.675385 15084 raft_consensus.cc:2258] T bf2a73282df74042a6885e7b2a93bb8f P b6b7fefcc60a4f6fae92bc19687153a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:15:55.678007 15084 ts_tablet_manager.cc:1905] T bf2a73282df74042a6885e7b2a93bb8f P b6b7fefcc60a4f6fae92bc19687153a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:15:55.679001 15085 ts_tablet_manager.cc:1918] T bf2a73282df74042a6885e7b2a93bb8f P 5274e26d9ee742fc9d5f93465ff450e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:15:55.679025 15086 ts_tablet_manager.cc:1918] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:15:55.679154 15085 log.cc:1196] T bf2a73282df74042a6885e7b2a93bb8f P 5274e26d9ee742fc9d5f93465ff450e8: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/bf2a73282df74042a6885e7b2a93bb8f
I20260926 15:15:55.679280 15086 log.cc:1196] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-0/wal/wals/bf2a73282df74042a6885e7b2a93bb8f
I20260926 15:15:55.679611 15085 ts_tablet_manager.cc:1939] T bf2a73282df74042a6885e7b2a93bb8f P 5274e26d9ee742fc9d5f93465ff450e8: Deleting consensus metadata
I20260926 15:15:55.679741 15086 ts_tablet_manager.cc:1939] T bf2a73282df74042a6885e7b2a93bb8f P 92e8f5d7dc5949468ab368fdfd2b2d2e: Deleting consensus metadata
I20260926 15:15:55.680465 14536 catalog_manager.cc:4841] TS 5274e26d9ee742fc9d5f93465ff450e8 (127.6.209.195:36427): tablet bf2a73282df74042a6885e7b2a93bb8f (table test-table [id=9d7398d17d484751a3b5a290b2b91b99]) successfully deleted
I20260926 15:15:55.680595 14535 catalog_manager.cc:4841] TS 92e8f5d7dc5949468ab368fdfd2b2d2e (127.6.209.193:40547): tablet bf2a73282df74042a6885e7b2a93bb8f (table test-table [id=9d7398d17d484751a3b5a290b2b91b99]) successfully deleted
I20260926 15:15:55.685336 15084 ts_tablet_manager.cc:1918] T bf2a73282df74042a6885e7b2a93bb8f P b6b7fefcc60a4f6fae92bc19687153a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:15:55.685539 15084 log.cc:1196] T bf2a73282df74042a6885e7b2a93bb8f P b6b7fefcc60a4f6fae92bc19687153a0: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/bf2a73282df74042a6885e7b2a93bb8f
I20260926 15:15:55.685979 15084 ts_tablet_manager.cc:1939] T bf2a73282df74042a6885e7b2a93bb8f P b6b7fefcc60a4f6fae92bc19687153a0: Deleting consensus metadata
I20260926 15:15:55.687214 14537 catalog_manager.cc:4841] TS b6b7fefcc60a4f6fae92bc19687153a0 (127.6.209.194:37611): tablet bf2a73282df74042a6885e7b2a93bb8f (table test-table [id=9d7398d17d484751a3b5a290b2b91b99]) successfully deleted
Sep 26 15:15:55 dist-test-slave-j6q7 krb5kdc[14506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.209.193@KRBTEST.COM
I20260926 15:15:55.694784 15058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:55.674880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56392 (local address 127.6.209.193:40547)
0926 15:15:55.675584 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:55.675590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:55.681098 (+ 5508us) server_negotiation.cc:407] Connection header received
0926 15:15:55.681294 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:55.681299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:55.681372 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:55.681494 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:55.682871 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:55.683657 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:55.685154 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:55.685615 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:55.685741 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:55.689707 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:55.689721 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:55.689745 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:55.689785 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:55.692224 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:55.693045 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:55.693051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:55.693053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:55.693152 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:55.693545 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:55.693548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:55.693550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:55.693894 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:55.694105 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:55.694393 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 15:15:55.694525 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603}
I20260926 15:15:55.695336 14752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:55.827639 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 14622
I20260926 15:15:55.850807 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 14761
Sep 26 15:15:55 dist-test-slave-j6q7 krb5kdc[14506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.209.194@KRBTEST.COM
I20260926 15:15:55.868060 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:55.851735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56244 (local address 127.6.209.194:37611)
0926 15:15:55.851899 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:55.851905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:55.851923 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:15:55.851982 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:55.851988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:55.852060 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:55.852174 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:55.853506 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:55.854553 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:55.856122 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:55.856680 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:55.856750 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:55.861643 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:55.861656 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:55.861674 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:55.861718 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:55.865212 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:55.865861 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:55.865869 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:55.865872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:55.866065 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:55.866426 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:55.866431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:55.866437 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:55.866788 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:55.867034 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:55.867518 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 15:15:55.867720 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 15:15:55.869652 14891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:55.992960 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 14761
I20260926 15:15:56.009248 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 14900
Sep 26 15:15:56 dist-test-slave-j6q7 krb5kdc[14506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.209.195@KRBTEST.COM
I20260926 15:15:56.022708 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:56.009866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46798 (local address 127.6.209.195:36427)
0926 15:15:56.010301 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:56.010305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:56.010319 (+ 14us) server_negotiation.cc:407] Connection header received
0926 15:15:56.010368 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:56.010371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:56.010433 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:56.010536 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:56.011777 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:56.012341 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:56.013487 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:56.013875 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:56.013956 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:56.017705 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:56.017718 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:56.017733 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:56.017758 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:56.020487 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:56.021112 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:56.021118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:56.021120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:56.021243 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:56.021532 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:56.021535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:56.021537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:56.021841 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:56.022104 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:56.022257 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 15:15:56.022384 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":245,"threads_started":1}
I20260926 15:15:56.023218 15030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:56.152230 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 14900
I20260926 15:15:56.174512 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 14519
I20260926 15:15:56.175398 14591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:15:56.277752 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 14519
2026-09-26T15:15:56Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5969 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:15:56 dist-test-slave-j6q7 krb5kdc[15165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:15:56 dist-test-slave-j6q7 krb5kdc[15165](info): set up 2 sockets
Sep 26 15:15:56 dist-test-slave-j6q7 krb5kdc[15165](info): commencing operation
krb5kdc: starting...
W20260926 15:15:57.269940 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:15:57 dist-test-slave-j6q7 krb5kdc[15165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:15:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:15:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:15:57.441561 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:34797
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:34797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:57.722054 15178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:57.722357 15178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:57.722522 15178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:57.729406 15178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:15:57.729552 15178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:57.729662 15178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:57.729763 15178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:15:57.729854 15178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:15:57.739039 15178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:34797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:34797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:57.740152 15178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:57.741391 15178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:57.752393 15183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:57.752456 15184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:57.752351 15186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:57.753082 15178 server_base.cc:1034] running on GCE node
I20260926 15:15:57.753729 15178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:57.754865 15178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:57.756062 15178 hybrid_clock.cc:648] HybridClock initialized: now 1790435757756025 us; error 31 us; skew 500 ppm
I20260926 15:15:57.756376 15178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T15:15:57Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1536 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:15:57 dist-test-slave-j6q7 krb5kdc[15192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:15:57 dist-test-slave-j6q7 krb5kdc[15192](info): set up 2 sockets
Sep 26 15:15:57 dist-test-slave-j6q7 krb5kdc[15192](info): commencing operation
krb5kdc: starting...
W20260926 15:15:58.801556 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:15:58 dist-test-slave-j6q7 krb5kdc[15192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:15:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:15:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:15:58.974124 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44707
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:40217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:44707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:59.251681 15205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:59.251952 15205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:59.252099 15205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:59.257889 15205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:15:59.258002 15205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:59.258100 15205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:59.258196 15205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:15:59.258251 15205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:15:59.267073 15205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:40217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:44707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:59.268123 15205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:59.269392 15205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:59.279109 15211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:59.279179 15210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:59.279155 15213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:59.280018 15205 server_base.cc:1034] running on GCE node
I20260926 15:15:59.280682 15205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:59.281850 15205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:59.283067 15205 hybrid_clock.cc:648] HybridClock initialized: now 1790435759283036 us; error 46 us; skew 500 ppm
I20260926 15:15:59.283396 15205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:59 dist-test-slave-j6q7 krb5kdc[15192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435759, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:59.286033 15205 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:15:59.287227 15205 webserver.cc:460] Webserver started at http://127.6.209.254:34701/ using document root <none> and password file <none>
I20260926 15:15:59.287776 15205 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:59.287899 15205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:59.288156 15205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:59.290752 15205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "ee8bd5ebff6b481a9035ffbf6baaa671"
format_stamp: "Formatted at 2026-09-26 15:15:59 on dist-test-slave-j6q7"
server_key: "82b17f77c85bac325ae32c7bd3e9cd54"
server_key_iv: "97d4fec930353a793d921e0d953647bd"
server_key_version: "encryptionkey@0"
I20260926 15:15:59.291399 15205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "ee8bd5ebff6b481a9035ffbf6baaa671"
format_stamp: "Formatted at 2026-09-26 15:15:59 on dist-test-slave-j6q7"
server_key: "82b17f77c85bac325ae32c7bd3e9cd54"
server_key_iv: "97d4fec930353a793d921e0d953647bd"
server_key_version: "encryptionkey@0"
I20260926 15:15:59.295729 15205 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 15:15:59.298570 15220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:59.299775 15205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 15:15:59.299939 15205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "ee8bd5ebff6b481a9035ffbf6baaa671"
format_stamp: "Formatted at 2026-09-26 15:15:59 on dist-test-slave-j6q7"
server_key: "82b17f77c85bac325ae32c7bd3e9cd54"
server_key_iv: "97d4fec930353a793d921e0d953647bd"
server_key_version: "encryptionkey@0"
I20260926 15:15:59.300107 15205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:59.315465 15205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:59.318490 15205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:59.318743 15205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:59.334445 15205 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:44707
I20260926 15:15:59.334462 15271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:44707 every 8 connection(s)
I20260926 15:15:59.335546 15205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:15:59.338646 15272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:15:59.340148 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 15205
I20260926 15:15:59.340343 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:15:59.341138 6983 external_mini_cluster.cc:1428] Setting key a89b555de271861870c90651f9c3e77e
I20260926 15:15:59.351015 15272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671: Bootstrap starting.
Sep 26 15:15:59 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:59.354130 15272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671: Neither blocks nor log segments found. Creating new log.
I20260926 15:15:59.355240 15272 log.cc:824] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671: Log is configured to *not* fsync() on all Append() calls
I20260926 15:15:59.357652 15272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671: No bootstrap required, opened a new log
I20260926 15:15:59.360775 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:59.342852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40336 (local address 127.6.209.254:44707)
0926 15:15:59.343516 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:59.343527 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:59.343560 (+ 33us) server_negotiation.cc:407] Connection header received
0926 15:15:59.344925 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:59.344945 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:59.345279 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:59.345561 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:59.347153 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:59.348267 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:59.350014 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:59.350528 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:59.350638 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:59.354479 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:59.354491 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:59.354507 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:59.354554 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:59.357339 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:59.357971 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:59.357976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:59.357983 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:59.358114 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:59.358473 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:59.358476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:59.358478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:59.358912 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:59.359115 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:59.359545 (+ 430us) server_negotiation.cc:299] Negotiation successful
0926 15:15:59.359858 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":325,"threads_started":1}
I20260926 15:15:59.365267 15272 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8bd5ebff6b481a9035ffbf6baaa671" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44707 } }
I20260926 15:15:59.365561 15272 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:15:59.365648 15272 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee8bd5ebff6b481a9035ffbf6baaa671, State: Initialized, Role: FOLLOWER
I20260926 15:15:59.366086 15272 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8bd5ebff6b481a9035ffbf6baaa671" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44707 } }
I20260926 15:15:59.366310 15272 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:15:59.366438 15272 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:15:59.366578 15272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:15:59.368443 15272 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8bd5ebff6b481a9035ffbf6baaa671" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44707 } }
I20260926 15:15:59.368923 15272 leader_election.cc:302] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee8bd5ebff6b481a9035ffbf6baaa671; no voters:
I20260926 15:15:59.369437 15272 leader_election.cc:288] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:15:59.369589 15277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:15:59.370468 15277 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [term 1 LEADER]: Becoming Leader. State: Replica: ee8bd5ebff6b481a9035ffbf6baaa671, State: Running, Role: LEADER
I20260926 15:15:59.371112 15272 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:15:59.370975 15277 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8bd5ebff6b481a9035ffbf6baaa671" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44707 } }
I20260926 15:15:59.374363 15279 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee8bd5ebff6b481a9035ffbf6baaa671. Latest consensus state: current_term: 1 leader_uuid: "ee8bd5ebff6b481a9035ffbf6baaa671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8bd5ebff6b481a9035ffbf6baaa671" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44707 } } }
I20260926 15:15:59.374836 15279 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:59.374472 15278 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee8bd5ebff6b481a9035ffbf6baaa671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8bd5ebff6b481a9035ffbf6baaa671" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44707 } } }
I20260926 15:15:59.375223 15278 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [sys.catalog]: This master's current role is: LEADER
I20260926 15:15:59.376777 15286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:15:59.380498 15286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:15:59.387825 15286 catalog_manager.cc:1347] Generated new cluster ID: 5e20a04d2aaa44569e5f790218a8db9d
I20260926 15:15:59.387914 15286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:15:59.398335 15286 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:15:59.399509 15286 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:15:59.411021 15286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671: Generated new TSK 0
I20260926 15:15:59.411628 15286 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
I20260926 15:15:59.453262 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44707
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:40217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:15:59.755618 15298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:15:59.755838 15298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:15:59.755980 15298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:15:59.762315 15298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:15:59.762398 15298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:15:59.762519 15298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:15:59.770738 15298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:40217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:15:59.771817 15298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:15:59.772994 15298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:15:59.784278 15303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:59.784339 15304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:15:59.784458 15306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:15:59.785365 15298 server_base.cc:1034] running on GCE node
I20260926 15:15:59.785880 15298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:15:59.786633 15298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:15:59.787843 15298 hybrid_clock.cc:648] HybridClock initialized: now 1790435759787804 us; error 51 us; skew 500 ppm
I20260926 15:15:59.788091 15298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:15:59 dist-test-slave-j6q7 krb5kdc[15192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435759, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:15:59.796030 15298 init.cc:377] Logged in from keytab as kudu/127.6.209.193@KRBTEST.COM (short username kudu)
I20260926 15:15:59.797260 15298 webserver.cc:460] Webserver started at http://127.6.209.193:34415/ using document root <none> and password file <none>
I20260926 15:15:59.797801 15298 fs_manager.cc:359] Metadata directory not provided
I20260926 15:15:59.797907 15298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:15:59.798173 15298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:15:59.800664 15298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "0b917b6fbb15405d87b42acb8f3f24de"
format_stamp: "Formatted at 2026-09-26 15:15:59 on dist-test-slave-j6q7"
server_key: "0c283b206747442bb88f6a26849acfa2"
server_key_iv: "114e63cf4e1ada0c5553596a62227382"
server_key_version: "encryptionkey@0"
I20260926 15:15:59.801483 15298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "0b917b6fbb15405d87b42acb8f3f24de"
format_stamp: "Formatted at 2026-09-26 15:15:59 on dist-test-slave-j6q7"
server_key: "0c283b206747442bb88f6a26849acfa2"
server_key_iv: "114e63cf4e1ada0c5553596a62227382"
server_key_version: "encryptionkey@0"
I20260926 15:15:59.805800 15298 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:15:59.808523 15313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:59.809698 15298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:15:59.809854 15298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "0b917b6fbb15405d87b42acb8f3f24de"
format_stamp: "Formatted at 2026-09-26 15:15:59 on dist-test-slave-j6q7"
server_key: "0c283b206747442bb88f6a26849acfa2"
server_key_iv: "114e63cf4e1ada0c5553596a62227382"
server_key_version: "encryptionkey@0"
I20260926 15:15:59.810034 15298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:15:59.828635 15298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:15:59.831918 15298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:15:59.832139 15298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:15:59.832986 15298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:15:59.834206 15298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:15:59.834282 15298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:59.834395 15298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:15:59.834447 15298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:15:59.862620 15298 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:38315
I20260926 15:15:59.862692 15425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:38315 every 8 connection(s)
I20260926 15:15:59.863639 15298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:15:59.871604 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 15298
I20260926 15:15:59.871795 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:15:59.872431 6983 external_mini_cluster.cc:1428] Setting key 2602110a4d6d6e0192a5400caeb0e588
Sep 26 15:15:59 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435759, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:15:59.882043 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:59.865751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:50217 (local address 127.6.209.254:44707)
0926 15:15:59.866121 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0926 15:15:59.866125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:15:59.866803 (+ 678us) server_negotiation.cc:407] Connection header received
0926 15:15:59.868366 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:15:59.868372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:15:59.868474 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:15:59.868612 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:15:59.869965 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:59.870584 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:59.872401 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:15:59.872911 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:15:59.873031 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:59.876406 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:15:59.876412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:15:59.876415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:15:59.876439 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:15:59.878700 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:59.879548 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:59.879551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:59.879554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:59.879709 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:15:59.880142 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:15:59.880148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:15:59.880152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:15:59.880464 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:15:59.880599 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 15:15:59.881208 (+ 609us) server_negotiation.cc:299] Negotiation successful
0926 15:15:59.881505 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":208,"threads_started":1}
I20260926 15:15:59.882519 15429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:15:59.866093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.193:50217)
0926 15:15:59.866663 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0926 15:15:59.866700 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 15:15:59.868132 (+ 1432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:15:59.868879 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:15:59.868887 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:15:59.869339 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:15:59.869824 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:59.869838 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:59.870724 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:59.870728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:59.872261 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:15:59.872275 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:15:59.873095 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:15:59.873101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:15:59.873325 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:15:59.873916 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:15:59.873934 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:15:59.876217 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:15:59.878829 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:59.878841 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:59.878857 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:59.879442 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:59.879823 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:15:59.879830 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:15:59.879836 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:15:59.880039 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:15:59.880570 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:15:59.880575 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:15:59.880851 (+ 276us) client_negotiation.cc:769] Sending connection context
0926 15:15:59.881051 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 15:15:59.881523 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":223,"threads_started":1}
I20260926 15:15:59.883759 15426 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44707
I20260926 15:15:59.884020 15426 heartbeater.cc:464] Registering TS with master...
I20260926 15:15:59.884737 15426 heartbeater.cc:511] Master 127.6.209.254:44707 requested a full tablet report, sending...
I20260926 15:15:59.886701 15237 ts_manager.cc:194] Registered new tserver with Master: 0b917b6fbb15405d87b42acb8f3f24de (127.6.209.193:38315)
I20260926 15:15:59.887890 15237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.193@KRBTEST.COM'} at 127.6.209.193:50217
WARNING: no policy specified for kudu/127.6.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
I20260926 15:15:59.914985 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44707
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:40217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:16:00.197851 15432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:16:00.198071 15432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:16:00.198236 15432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:16:00.204782 15432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:16:00.204893 15432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:16:00.205060 15432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:16:00.213047 15432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:40217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:16:00.214030 15432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:16:00.215059 15432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:16:00.226454 15438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:00.226477 15437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:00.226694 15440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:16:00.227057 15432 server_base.cc:1034] running on GCE node
I20260926 15:16:00.227578 15432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:16:00.228359 15432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:16:00.229540 15432 hybrid_clock.cc:648] HybridClock initialized: now 1790435760229526 us; error 42 us; skew 500 ppm
I20260926 15:16:00.229854 15432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:16:00 dist-test-slave-j6q7 krb5kdc[15192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435760, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:16:00.232542 15432 init.cc:377] Logged in from keytab as kudu/127.6.209.194@KRBTEST.COM (short username kudu)
I20260926 15:16:00.233955 15432 webserver.cc:460] Webserver started at http://127.6.209.194:37235/ using document root <none> and password file <none>
I20260926 15:16:00.234545 15432 fs_manager.cc:359] Metadata directory not provided
I20260926 15:16:00.234675 15432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:16:00.235019 15432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:16:00.237705 15432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "71100a1311e74d85925aef1a75c83d50"
format_stamp: "Formatted at 2026-09-26 15:16:00 on dist-test-slave-j6q7"
server_key: "be58c6256d0253449a24039997e8347b"
server_key_iv: "a27ec3ce7881bafdd64829b145f2f1b8"
server_key_version: "encryptionkey@0"
I20260926 15:16:00.238430 15432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "71100a1311e74d85925aef1a75c83d50"
format_stamp: "Formatted at 2026-09-26 15:16:00 on dist-test-slave-j6q7"
server_key: "be58c6256d0253449a24039997e8347b"
server_key_iv: "a27ec3ce7881bafdd64829b145f2f1b8"
server_key_version: "encryptionkey@0"
I20260926 15:16:00.242992 15432 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 15:16:00.245849 15447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:00.246866 15432 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:16:00.247009 15432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "71100a1311e74d85925aef1a75c83d50"
format_stamp: "Formatted at 2026-09-26 15:16:00 on dist-test-slave-j6q7"
server_key: "be58c6256d0253449a24039997e8347b"
server_key_iv: "a27ec3ce7881bafdd64829b145f2f1b8"
server_key_version: "encryptionkey@0"
I20260926 15:16:00.247181 15432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:16:00.258490 15432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:16:00.262279 15432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:16:00.262528 15432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:16:00.263288 15432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:16:00.264325 15432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:16:00.264413 15432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:00.264537 15432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:16:00.264601 15432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:00.292603 15432 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:34683
I20260926 15:16:00.292666 15559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:34683 every 8 connection(s)
I20260926 15:16:00.293602 15432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:16:00.299994 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 15432
I20260926 15:16:00.300197 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:16:00.300580 6983 external_mini_cluster.cc:1428] Setting key 9472ec0f4728796eb00e29b3bdc21e51
Sep 26 15:16:00 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435760, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:16:00.312091 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:00.295523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:35309 (local address 127.6.209.254:44707)
0926 15:16:00.295675 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:00.295678 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:00.296621 (+ 943us) server_negotiation.cc:407] Connection header received
0926 15:16:00.298272 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:00.298278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:00.298361 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:00.298470 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:00.299758 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:00.300534 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:00.302204 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:00.302727 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:00.302867 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:00.306123 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:00.306130 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:00.306137 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:16:00.306176 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:00.309429 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:00.310236 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:00.310239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:00.310241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:00.310344 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:00.310742 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:00.310744 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:00.310746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:00.310943 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:00.311032 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:00.311715 (+ 683us) server_negotiation.cc:299] Negotiation successful
0926 15:16:00.311854 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 15:16:00.313405 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:00.295929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.194:35309)
0926 15:16:00.296498 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:00.296527 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:00.297993 (+ 1466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:00.298677 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:00.298684 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:00.299113 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:00.299599 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:00.299614 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:00.300668 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:00.300674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:00.302050 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:00.302061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:00.302961 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:00.302966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:00.303142 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:00.303715 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:00.303734 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:00.305923 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:16:00.309566 (+ 3643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:16:00.309575 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:16:00.309590 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:16:00.310135 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:16:00.310435 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:16:00.310441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:16:00.310446 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:16:00.310640 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:16:00.311042 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:16:00.311052 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:16:00.311386 (+ 334us) client_negotiation.cc:769] Sending connection context
0926 15:16:00.311641 (+ 255us) client_negotiation.cc:240] Negotiation successful
0926 15:16:00.312124 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":234,"threads_started":1}
I20260926 15:16:00.314540 15560 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44707
I20260926 15:16:00.314806 15560 heartbeater.cc:464] Registering TS with master...
I20260926 15:16:00.315407 15560 heartbeater.cc:511] Master 127.6.209.254:44707 requested a full tablet report, sending...
I20260926 15:16:00.316540 15237 ts_manager.cc:194] Registered new tserver with Master: 71100a1311e74d85925aef1a75c83d50 (127.6.209.194:34683)
I20260926 15:16:00.317359 15237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.194@KRBTEST.COM'} at 127.6.209.194:35309
WARNING: no policy specified for kudu/127.6.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
I20260926 15:16:00.343503 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44707
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:40217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:16:00.628361 15565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:16:00.628670 15565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:16:00.628877 15565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:16:00.635372 15565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:16:00.635490 15565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:16:00.635684 15565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:16:00.647251 15565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:40217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:16:00.648377 15565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:16:00.649722 15565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:16:00.661587 15571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:00.661677 15573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:00.661639 15570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:16:00.662796 15565 server_base.cc:1034] running on GCE node
I20260926 15:16:00.663273 15565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:16:00.664032 15565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:16:00.665208 15565 hybrid_clock.cc:648] HybridClock initialized: now 1790435760665179 us; error 27 us; skew 500 ppm
I20260926 15:16:00.665433 15565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:16:00 dist-test-slave-j6q7 krb5kdc[15192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435760, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:16:00.668058 15565 init.cc:377] Logged in from keytab as kudu/127.6.209.195@KRBTEST.COM (short username kudu)
I20260926 15:16:00.669330 15565 webserver.cc:460] Webserver started at http://127.6.209.195:44883/ using document root <none> and password file <none>
I20260926 15:16:00.669889 15565 fs_manager.cc:359] Metadata directory not provided
I20260926 15:16:00.669978 15565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:16:00.670212 15565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:16:00.672977 15565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "65c787abba24463980b4ac76ad3514e6"
format_stamp: "Formatted at 2026-09-26 15:16:00 on dist-test-slave-j6q7"
server_key: "1afeb6f20f254f994d29102301adbf1f"
server_key_iv: "d6e00009197edf51da2f80a2d675f94b"
server_key_version: "encryptionkey@0"
I20260926 15:16:00.673657 15565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "65c787abba24463980b4ac76ad3514e6"
format_stamp: "Formatted at 2026-09-26 15:16:00 on dist-test-slave-j6q7"
server_key: "1afeb6f20f254f994d29102301adbf1f"
server_key_iv: "d6e00009197edf51da2f80a2d675f94b"
server_key_version: "encryptionkey@0"
I20260926 15:16:00.678455 15565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260926 15:16:00.681464 15580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:00.682596 15565 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260926 15:16:00.682714 15565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "65c787abba24463980b4ac76ad3514e6"
format_stamp: "Formatted at 2026-09-26 15:16:00 on dist-test-slave-j6q7"
server_key: "1afeb6f20f254f994d29102301adbf1f"
server_key_iv: "d6e00009197edf51da2f80a2d675f94b"
server_key_version: "encryptionkey@0"
I20260926 15:16:00.682852 15565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:16:00.699266 15565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:16:00.702661 15565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:16:00.702919 15565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:16:00.703706 15565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:16:00.704790 15565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:16:00.704909 15565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:00.705060 15565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:16:00.705109 15565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:00.735299 15565 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:46669
I20260926 15:16:00.735361 15692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:46669 every 8 connection(s)
I20260926 15:16:00.736478 15565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:16:00.740136 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 15565
I20260926 15:16:00.740331 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:16:00.740706 6983 external_mini_cluster.cc:1428] Setting key 30d49cd8250f65b367033a092b879535
Sep 26 15:16:00 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435760, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:16:00.754536 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:00.738755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:35805 (local address 127.6.209.254:44707)
0926 15:16:00.738917 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:00.738921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:00.739972 (+ 1051us) server_negotiation.cc:407] Connection header received
0926 15:16:00.741919 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:00.741924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:00.742014 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:00.742124 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:00.743693 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:00.744348 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:00.745751 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:00.746146 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:00.746279 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:00.749555 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:00.749560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:00.749563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:16:00.749592 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:00.751874 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:00.752487 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:00.752492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:00.752501 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:00.752613 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:00.753058 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:00.753062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:00.753065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:00.753303 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:00.753445 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:00.754027 (+ 582us) server_negotiation.cc:299] Negotiation successful
0926 15:16:00.754242 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:16:00.755486 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:00.739133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.195:35805)
0926 15:16:00.739824 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:00.739866 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:00.741622 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:00.742335 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:00.742343 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:00.742947 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:00.743542 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:00.743556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:00.744501 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:00.744505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:00.745620 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:00.745630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:00.746361 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:00.746368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:00.746585 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:00.747319 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:00.747342 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:00.749360 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:16:00.751997 (+ 2637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:16:00.752005 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:16:00.752017 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:16:00.752372 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:16:00.752727 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:16:00.752731 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:16:00.752735 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:16:00.752951 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:16:00.753419 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:16:00.753425 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:16:00.753724 (+ 299us) client_negotiation.cc:769] Sending connection context
0926 15:16:00.753946 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 15:16:00.754379 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":250,"threads_started":1}
I20260926 15:16:00.756634 15693 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44707
I20260926 15:16:00.756935 15693 heartbeater.cc:464] Registering TS with master...
I20260926 15:16:00.757551 15693 heartbeater.cc:511] Master 127.6.209.254:44707 requested a full tablet report, sending...
I20260926 15:16:00.758757 15237 ts_manager.cc:194] Registered new tserver with Master: 65c787abba24463980b4ac76ad3514e6 (127.6.209.195:46669)
I20260926 15:16:00.759462 15237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.195@KRBTEST.COM'} at 127.6.209.195:35805
I20260926 15:16:00.767074 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.193.keytab.
Entry for principal oryx/127.6.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.193.keytab.
WARNING: no policy specified for oryx/127.6.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.194.keytab.
Entry for principal oryx/127.6.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.194.keytab.
WARNING: no policy specified for oryx/127.6.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.195.keytab.
Entry for principal oryx/127.6.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.195.keytab.
I20260926 15:16:00.875662 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 15298
Sep 26 15:16:00 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.193@KRBTEST.COM
I20260926 15:16:00.890256 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:00.876629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50844 (local address 127.6.209.193:38315)
0926 15:16:00.877568 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:00.877575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:00.877599 (+ 24us) server_negotiation.cc:407] Connection header received
0926 15:16:00.877652 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:00.877659 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:00.877847 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:00.877969 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:00.879206 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:00.880123 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:00.881338 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:00.881849 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:00.881956 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:00.885510 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:00.885520 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:00.885533 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:16:00.885563 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:00.887992 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:00.888550 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:00.888556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:00.888558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:00.888696 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:00.889044 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:00.889048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:00.889050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:00.889351 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:00.889556 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:00.889751 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 15:16:00.889932 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":515,"threads_started":1}
I20260926 15:16:00.891114 15426 heartbeater.cc:503] Master 127.6.209.254:44707 was elected leader, sending a full tablet report...
I20260926 15:16:00.891494 15421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:16:00.967919 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 15298
I20260926 15:16:00.983001 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 15432
Sep 26 15:16:00 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.194@KRBTEST.COM
I20260926 15:16:00.998584 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:00.984013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47156 (local address 127.6.209.194:34683)
0926 15:16:00.984760 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:00.984771 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:00.984799 (+ 28us) server_negotiation.cc:407] Connection header received
0926 15:16:00.984886 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:00.984896 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:00.985237 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:00.985390 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:00.986975 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:00.988070 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:00.989328 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:00.989857 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:00.989945 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:00.993641 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:00.993651 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:00.993664 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:16:00.993693 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:00.996386 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:00.996982 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:00.996988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:00.996990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:00.997097 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:00.997391 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:00.997394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:00.997396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:00.997686 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:00.997896 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:00.998123 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 15:16:00.998299 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":429,"threads_started":1}
I20260926 15:16:00.999792 15555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:16:01.073151 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 15432
I20260926 15:16:01.087168 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 15565
Sep 26 15:16:01 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.195@KRBTEST.COM
I20260926 15:16:01.101640 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:01.087887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41404 (local address 127.6.209.195:46669)
0926 15:16:01.088450 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:01.088457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:01.088481 (+ 24us) server_negotiation.cc:407] Connection header received
0926 15:16:01.088533 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:01.088539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:01.088762 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:01.088945 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:01.090280 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:01.091144 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:01.092170 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:01.092635 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:01.092714 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:01.096537 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:01.096546 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:01.096558 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:16:01.096583 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:01.099318 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:01.099900 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:01.099907 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:01.099909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:01.100025 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:01.100301 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:01.100304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:01.100307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:01.100587 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:01.100892 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:01.101190 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 15:16:01.101362 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":290,"threads_started":1}
I20260926 15:16:01.102798 15688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:16:01.177222 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 15565
I20260926 15:16:01.191758 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 15205
I20260926 15:16:01.192787 15267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:16:01.283519 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 15205
I20260926 15:16:01.302071 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44707
--webserver_interface=127.6.209.254
--webserver_port=34701
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:40217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:44707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:16:01.591734 15710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:16:01.592054 15710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:16:01.592216 15710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:16:01.599332 15710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:16:01.599481 15710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:16:01.599587 15710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:16:01.599694 15710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:16:01.599784 15710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:16:01.609297 15710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:40217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:44707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=34701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:16:01.610579 15710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:16:01.612079 15710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:16:01.623291 15715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:01.623356 15716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:01.623391 15718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:16:01.624562 15710 server_base.cc:1034] running on GCE node
I20260926 15:16:01.625342 15710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:16:01.626579 15710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:16:01.627794 15710 hybrid_clock.cc:648] HybridClock initialized: now 1790435761627750 us; error 65 us; skew 500 ppm
I20260926 15:16:01.628111 15710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:16:01 dist-test-slave-j6q7 krb5kdc[15192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435761, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:16:01.630924 15710 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:16:01.632264 15710 webserver.cc:460] Webserver started at http://127.6.209.254:34701/ using document root <none> and password file <none>
I20260926 15:16:01.632994 15710 fs_manager.cc:359] Metadata directory not provided
I20260926 15:16:01.633136 15710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:16:01.638743 15710 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 15:16:01.641368 15725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:01.642560 15710 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:16:01.642706 15710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "ee8bd5ebff6b481a9035ffbf6baaa671"
format_stamp: "Formatted at 2026-09-26 15:15:59 on dist-test-slave-j6q7"
server_key: "82b17f77c85bac325ae32c7bd3e9cd54"
server_key_iv: "97d4fec930353a793d921e0d953647bd"
server_key_version: "encryptionkey@0"
I20260926 15:16:01.643539 15710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:16:01.659432 15710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:16:01.663010 15710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:16:01.663210 15710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:16:01.680663 15710 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:44707
I20260926 15:16:01.680694 15776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:44707 every 8 connection(s)
I20260926 15:16:01.682220 15710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:16:01.688414 15777 sys_catalog.cc:263] Verifying existing consensus state
I20260926 15:16:01.690515 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 15710
I20260926 15:16:01.691412 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:38315
--local_ip_for_outbound_sockets=127.6.209.193
--tserver_master_addrs=127.6.209.254:44707
--webserver_port=34415
--webserver_interface=127.6.209.193
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:40217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.209.193
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:16:01.692166 15777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671: Bootstrap starting.
I20260926 15:16:01.711884 15777 log.cc:824] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671: Log is configured to *not* fsync() on all Append() calls
I20260926 15:16:01.720118 15777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:16:01.720607 15777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671: Bootstrap complete.
I20260926 15:16:01.729269 15777 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8bd5ebff6b481a9035ffbf6baaa671" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44707 } }
I20260926 15:16:01.730054 15777 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee8bd5ebff6b481a9035ffbf6baaa671, State: Initialized, Role: FOLLOWER
I20260926 15:16:01.730551 15777 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8bd5ebff6b481a9035ffbf6baaa671" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44707 } }
I20260926 15:16:01.730763 15777 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:16:01.730898 15777 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:16:01.731063 15777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [term 1 FOLLOWER]: Advancing to term 2
I20260926 15:16:01.734673 15777 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8bd5ebff6b481a9035ffbf6baaa671" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44707 } }
I20260926 15:16:01.735113 15777 leader_election.cc:302] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee8bd5ebff6b481a9035ffbf6baaa671; no voters:
I20260926 15:16:01.735759 15777 leader_election.cc:288] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 15:16:01.735884 15781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 15:16:01.737269 15781 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [term 2 LEADER]: Becoming Leader. State: Replica: ee8bd5ebff6b481a9035ffbf6baaa671, State: Running, Role: LEADER
I20260926 15:16:01.737771 15781 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8bd5ebff6b481a9035ffbf6baaa671" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44707 } }
I20260926 15:16:01.737975 15777 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:16:01.740185 15783 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee8bd5ebff6b481a9035ffbf6baaa671. Latest consensus state: current_term: 2 leader_uuid: "ee8bd5ebff6b481a9035ffbf6baaa671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8bd5ebff6b481a9035ffbf6baaa671" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44707 } } }
I20260926 15:16:01.740615 15783 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [sys.catalog]: This master's current role is: LEADER
I20260926 15:16:01.741266 15782 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ee8bd5ebff6b481a9035ffbf6baaa671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8bd5ebff6b481a9035ffbf6baaa671" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44707 } } }
I20260926 15:16:01.742013 15782 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671 [sys.catalog]: This master's current role is: LEADER
I20260926 15:16:01.743469 15787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:16:01.749603 15787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:16:01.750972 15787 catalog_manager.cc:1259] Loaded cluster ID: 5e20a04d2aaa44569e5f790218a8db9d
I20260926 15:16:01.751124 15787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:16:01.754928 15787 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:16:01.756359 15787 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ee8bd5ebff6b481a9035ffbf6baaa671: Loaded TSK: 0
I20260926 15:16:01.757953 15787 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 15:16:02.007627 15779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:16:02.007923 15779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:16:02.008111 15779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:16:02.014647 15779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:16:02.014783 15779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:16:02.014933 15779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:16:02.023741 15779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:40217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.193.keytab
--principal=oryx/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:38315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=34415
--tserver_master_addrs=127.6.209.254:44707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:16:02.024895 15779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:16:02.026026 15779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:16:02.040064 15806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:02.040055 15804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:02.041389 15803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:16:02.042546 15779 server_base.cc:1034] running on GCE node
I20260926 15:16:02.043072 15779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:16:02.043933 15779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:16:02.045286 15779 hybrid_clock.cc:648] HybridClock initialized: now 1790435762045195 us; error 41 us; skew 500 ppm
I20260926 15:16:02.045612 15779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:16:02 dist-test-slave-j6q7 krb5kdc[15192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435762, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:16:02.048702 15779 init.cc:377] Logged in from keytab as oryx/127.6.209.193@KRBTEST.COM (short username oryx)
I20260926 15:16:02.050194 15779 webserver.cc:460] Webserver started at http://127.6.209.193:34415/ using document root <none> and password file <none>
I20260926 15:16:02.050760 15779 fs_manager.cc:359] Metadata directory not provided
I20260926 15:16:02.050874 15779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:16:02.055630 15779 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 15:16:02.058120 15813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:02.059412 15779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:16:02.059535 15779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "0b917b6fbb15405d87b42acb8f3f24de"
format_stamp: "Formatted at 2026-09-26 15:15:59 on dist-test-slave-j6q7"
server_key: "0c283b206747442bb88f6a26849acfa2"
server_key_iv: "114e63cf4e1ada0c5553596a62227382"
server_key_version: "encryptionkey@0"
I20260926 15:16:02.060391 15779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:16:02.072796 15779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:16:02.076505 15779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:16:02.076682 15779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:16:02.077482 15779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:16:02.078497 15779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:16:02.078577 15779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:02.078642 15779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:16:02.078708 15779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:02.107741 15779 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:38315
I20260926 15:16:02.107933 15925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:38315 every 8 connection(s)
I20260926 15:16:02.109133 15779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:16:02.113129 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 15779
I20260926 15:16:02.113982 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:34683
--local_ip_for_outbound_sockets=127.6.209.194
--tserver_master_addrs=127.6.209.254:44707
--webserver_port=37235
--webserver_interface=127.6.209.194
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:40217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.209.194
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:16:02 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:02 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 15:16:02.408330 15930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:16:02.408610 15930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:16:02.408776 15930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:16:02.414942 15930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:16:02.415066 15930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:16:02.415206 15930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:16:02.423610 15930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:40217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.194.keytab
--principal=oryx/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:34683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=37235
--tserver_master_addrs=127.6.209.254:44707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:16:02.424612 15930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:16:02.425746 15930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:16:02.441645 15937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:02.441854 15936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:02.442415 15939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:16:02.442865 15930 server_base.cc:1034] running on GCE node
I20260926 15:16:02.443854 15930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:16:02.445351 15930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:16:02.446641 15930 hybrid_clock.cc:648] HybridClock initialized: now 1790435762446561 us; error 48 us; skew 500 ppm
I20260926 15:16:02.447187 15930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:16:02 dist-test-slave-j6q7 krb5kdc[15192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435762, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:16:02.452222 15930 init.cc:377] Logged in from keytab as oryx/127.6.209.194@KRBTEST.COM (short username oryx)
I20260926 15:16:02.454460 15930 webserver.cc:460] Webserver started at http://127.6.209.194:37235/ using document root <none> and password file <none>
I20260926 15:16:02.455525 15930 fs_manager.cc:359] Metadata directory not provided
I20260926 15:16:02.455716 15930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:16:02.464727 15930 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.003s
I20260926 15:16:02.468604 15946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:02.470569 15930 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 15:16:02.470785 15930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "71100a1311e74d85925aef1a75c83d50"
format_stamp: "Formatted at 2026-09-26 15:16:00 on dist-test-slave-j6q7"
server_key: "be58c6256d0253449a24039997e8347b"
server_key_iv: "a27ec3ce7881bafdd64829b145f2f1b8"
server_key_version: "encryptionkey@0"
I20260926 15:16:02.472579 15930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:16:02.493549 15930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:16:02.497578 15930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:16:02.497810 15930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:16:02.498720 15930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:16:02.500020 15930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:16:02.500119 15930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:02.500252 15930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:16:02.500329 15930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:02.536304 15930 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:34683
I20260926 15:16:02.536353 16058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:34683 every 8 connection(s)
I20260926 15:16:02.537707 15930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:16:02.544907 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 15930
I20260926 15:16:02.545884 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:46669
--local_ip_for_outbound_sockets=127.6.209.195
--tserver_master_addrs=127.6.209.254:44707
--webserver_port=44883
--webserver_interface=127.6.209.195
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:40217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.209.195
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:16:02 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:02 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 15:16:02.858965 16063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:16:02.859246 16063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:16:02.859411 16063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:16:02.865908 16063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:16:02.866044 16063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:16:02.866197 16063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:16:02.874416 16063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:40217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/krb5kdc/oryx_127.6.209.195.keytab
--principal=oryx/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:46669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=44883
--tserver_master_addrs=127.6.209.254:44707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:16:02.875422 16063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:16:02.876461 16063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:16:02.892202 16070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:02.897477 16069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:02.898729 16072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:16:02.899194 16063 server_base.cc:1034] running on GCE node
I20260926 15:16:02.899791 16063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:16:02.900676 16063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:16:02.902012 16063 hybrid_clock.cc:648] HybridClock initialized: now 1790435762901962 us; error 65 us; skew 500 ppm
I20260926 15:16:02.902281 16063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:16:02 dist-test-slave-j6q7 krb5kdc[15192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435762, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:16:02.905251 16063 init.cc:377] Logged in from keytab as oryx/127.6.209.195@KRBTEST.COM (short username oryx)
I20260926 15:16:02.909932 16063 webserver.cc:460] Webserver started at http://127.6.209.195:44883/ using document root <none> and password file <none>
I20260926 15:16:02.910566 16063 fs_manager.cc:359] Metadata directory not provided
I20260926 15:16:02.910701 16063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:16:02.915273 16063 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:16:02.917974 16079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:02.919222 16063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:16:02.919389 16063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "65c787abba24463980b4ac76ad3514e6"
format_stamp: "Formatted at 2026-09-26 15:16:00 on dist-test-slave-j6q7"
server_key: "1afeb6f20f254f994d29102301adbf1f"
server_key_iv: "d6e00009197edf51da2f80a2d675f94b"
server_key_version: "encryptionkey@0"
I20260926 15:16:02.920229 16063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:16:02.939446 16063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:16:02.943449 16063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:16:02.943698 16063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:16:02.944628 16063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:16:02.945833 16063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:16:02.945935 16063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:02.946053 16063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:16:02.946164 16063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260926 15:16:02.992241 16191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:46669 every 8 connection(s)
I20260926 15:16:02.992887 16063 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:46669
I20260926 15:16:02.994737 16063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:16:03.004654 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 16063
Sep 26 15:16:03 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:03 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:03.037815 16196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:03.005455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40522 (local address 127.6.209.254:44707)
0926 15:16:03.016024 (+ 10569us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:03.016031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:03.016333 (+ 302us) server_negotiation.cc:407] Connection header received
0926 15:16:03.016512 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:03.016517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:03.016578 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:03.016661 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:03.019737 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:03.021138 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:03.023145 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:03.023714 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:03.023892 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:03.028130 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:03.028141 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:03.028159 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:16:03.028204 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:03.031974 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:03.032740 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:03.032749 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:03.032758 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:03.032993 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:03.033829 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:03.033877 (+ 48us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:03.033882 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:03.034422 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:03.034704 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:03.036426 (+ 1722us) server_negotiation.cc:299] Negotiation successful
0926 15:16:03.036853 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10452,"thread_start_us":10375,"threads_started":1}
W20260926 15:16:03.549685 15928 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.427s user 0.000s sys 0.016s
W20260926 15:16:03.551182 15929 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:02.111788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:50119 (local address 127.6.209.254:44707)
0926 15:16:02.115531 (+ 3743us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:02.115540 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:02.115571 (+ 31us) server_negotiation.cc:407] Connection header received
0926 15:16:02.115842 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:02.115866 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:02.116141 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:02.116450 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:02.117823 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:02.118988 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:02.120458 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:02.121137 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:02.121279 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:03.550931 (+1429652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.193:50119: BlockingRecv error: recv got EOF from 127.6.209.193:50119 (error 108)
Metrics: {"server-negotiator.queue_time_us":3574,"thread_start_us":3416,"threads_started":1}
I20260926 15:16:03.551862 15928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:02.111790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.193:50119)
0926 15:16:02.113428 (+ 1638us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:02.113462 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:02.114965 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:02.116598 (+ 1633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:02.116607 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:02.117144 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:02.117633 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:02.117646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:02.119150 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:02.119155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:02.120327 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:02.120343 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:02.121354 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:02.121363 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:02.121680 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:02.122353 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:02.122382 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:03.550744 (+1428362us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":286,"threads_started":1}
W20260926 15:16:03.553117 15926 heartbeater.cc:644] Failed to heartbeat to 127.6.209.254:44707 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.209.254:44707: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Sep 26 15:16:03 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:03 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:03.568403 15928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:03.554011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.193:48601)
0926 15:16:03.554264 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:03.554280 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:03.554426 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:03.554870 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:03.554877 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:03.555438 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:03.555713 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:03.555727 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:03.557489 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:03.557497 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:03.559604 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:03.559624 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:03.561099 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:03.561108 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:03.561376 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:03.562296 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:03.562337 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:03.567974 (+ 5637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152}
W20260926 15:16:03.568614 15929 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:03.554103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:48601 (local address 127.6.209.254:44707)
0926 15:16:03.554261 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:03.554267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:03.554368 (+ 101us) server_negotiation.cc:407] Connection header received
0926 15:16:03.554568 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:03.554575 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:03.554685 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:03.554805 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:03.555881 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:03.557329 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:03.559838 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:03.560808 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:03.560959 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:03.568246 (+ 7287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.193:48601: BlockingRecv error: recv got EOF from 127.6.209.193:48601 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 15:16:03 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:03 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 15:16:03.584448 15929 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:03.569865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:35817 (local address 127.6.209.254:44707)
0926 15:16:03.570353 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:03.570361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:03.570780 (+ 419us) server_negotiation.cc:407] Connection header received
0926 15:16:03.571327 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:03.571334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:03.571447 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:03.571550 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:03.573021 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:03.574421 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:03.576648 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:03.577503 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:03.577655 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:03.584165 (+ 6510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.193:35817: BlockingRecv error: recv got EOF from 127.6.209.193:35817 (error 108)
Metrics: {"server-negotiator.queue_time_us":350}
I20260926 15:16:03.585059 15928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:03.569747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.193:35817)
0926 15:16:03.570522 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:03.570543 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:03.570879 (+ 336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:03.572069 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:03.572076 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:03.572580 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:03.572851 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:03.572865 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:03.574590 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:03.574597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:03.576474 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:03.576504 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:03.577677 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:03.577686 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:03.577945 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:03.578801 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:03.578830 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:03.584692 (+ 5862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":635}
W20260926 15:16:03.585404 15926 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 15:16:03.884359 16062 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.334s user 0.001s sys 0.011s
W20260926 15:16:03.885912 16061 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:02.541434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:55319 (local address 127.6.209.254:44707)
0926 15:16:02.542078 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:02.542088 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:02.542781 (+ 693us) server_negotiation.cc:407] Connection header received
0926 15:16:02.544422 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:02.544426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:02.544499 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:02.544642 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:02.546129 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:02.546900 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:02.548393 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:02.548784 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:02.548972 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:03.885686 (+1336714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.194:55319: BlockingRecv error: recv got EOF from 127.6.209.194:55319 (error 108)
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":281,"threads_started":1}
I20260926 15:16:03.887068 16062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:02.541922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.194:55319)
0926 15:16:02.542631 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:02.542668 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:02.544205 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:02.544835 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:02.544846 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:02.545400 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:02.545931 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:02.545946 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:02.547026 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:02.547030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:02.548277 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:02.548286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:02.549056 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:02.549065 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:02.549287 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:02.549945 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:02.549964 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:03.885558 (+1335594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":270,"threads_started":1}
W20260926 15:16:03.887970 16059 heartbeater.cc:644] Failed to heartbeat to 127.6.209.254:44707 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.209.254:44707: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Sep 26 15:16:03 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:03 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 15:16:03.898422 16061 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:03.888710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:37163 (local address 127.6.209.254:44707)
0926 15:16:03.888909 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:03.888917 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:03.888937 (+ 20us) server_negotiation.cc:407] Connection header received
0926 15:16:03.889115 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:03.889120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:03.889197 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:03.889310 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:03.890098 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:03.891725 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:03.892783 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:03.893602 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:03.893760 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:03.898285 (+ 4525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.194:37163: BlockingRecv error: recv got EOF from 127.6.209.194:37163 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 15:16:03.898550 16062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:03.888500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.194:37163)
0926 15:16:03.888736 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:03.888761 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:03.888970 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:03.889338 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:03.889342 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:03.889735 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:03.889922 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:03.889934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:03.891865 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:03.891869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:03.892658 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:03.892667 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:03.893766 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:03.893770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:03.893870 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:03.894397 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:03.894411 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:03.898172 (+ 3761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
Sep 26 15:16:03 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:03 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:03.909509 16062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:03.899289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.194:60363)
0926 15:16:03.899589 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:03.899612 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:03.899745 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:03.900083 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:03.900091 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:03.900608 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:03.900927 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:03.900956 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:03.902080 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:03.902087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:03.903996 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:03.904011 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:03.904603 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:03.904611 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:03.904893 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:03.905464 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:03.905492 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:03.909259 (+ 3767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147}
W20260926 15:16:03.909749 16059 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 15:16:03.909904 16061 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:03.899391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:60363 (local address 127.6.209.254:44707)
0926 15:16:03.899523 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:03.899527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:03.899687 (+ 160us) server_negotiation.cc:407] Connection header received
0926 15:16:03.899854 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:03.899857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:03.899915 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:03.899988 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:03.901166 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:03.901950 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:03.904137 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:03.904455 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:03.904531 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:03.909414 (+ 4883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.194:60363: BlockingRecv error: recv got EOF from 127.6.209.194:60363 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 15:16:04.045481 16194 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.022s user 0.006s sys 0.009s
W20260926 15:16:04.046586 16195 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:02.999116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:58557 (local address 127.6.209.254:44707)
0926 15:16:03.005168 (+ 6052us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:03.005174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:03.012864 (+ 7690us) server_negotiation.cc:407] Connection header received
0926 15:16:03.012963 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:03.012968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:03.013083 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:03.013258 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:03.017897 (+ 4639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:03.019055 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:03.021277 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:03.021830 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:03.021919 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:04.046311 (+1024392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.195:58557: BlockingRecv error: recv got EOF from 127.6.209.195:58557 (error 108)
Metrics: {"server-negotiator.queue_time_us":5924,"thread_start_us":5810,"threads_started":1}
I20260926 15:16:04.047043 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:02.998477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.195:58557)
0926 15:16:03.005383 (+ 6906us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:03.005454 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:03.008974 (+ 3520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:03.016776 (+ 7802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:03.016785 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:03.017271 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:03.017743 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:03.017757 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:03.019244 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:03.019249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:03.021149 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:03.021160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:03.022209 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:03.022214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:03.022451 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:03.023083 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:03.023118 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:04.046142 (+1023024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":6553,"thread_start_us":6405,"threads_started":1}
W20260926 15:16:04.047886 16192 heartbeater.cc:644] Failed to heartbeat to 127.6.209.254:44707 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.209.254:44707: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Sep 26 15:16:04 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:04 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:04.055948 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:04.048435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.195:35005)
0926 15:16:04.048676 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:04.048692 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:04.048776 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:04.049098 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:04.049101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:04.049407 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:04.049551 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:04.049558 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:04.050452 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:04.050455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:04.051340 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:04.051350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:04.051937 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:04.051940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:04.052040 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:04.052530 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:04.052545 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:04.055680 (+ 3135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84}
W20260926 15:16:04.056064 16195 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:04.048568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:35005 (local address 127.6.209.254:44707)
0926 15:16:04.048723 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:04.048728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:04.048743 (+ 15us) server_negotiation.cc:407] Connection header received
0926 15:16:04.048929 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:04.048932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:04.048988 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:04.049064 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:04.049660 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:04.050351 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:04.051443 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:04.051814 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:04.051892 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:04.055840 (+ 3948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.195:35005: BlockingRecv error: recv got EOF from 127.6.209.195:35005 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 15:16:04 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:04 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:04.063769 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:04.056609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.195:50973)
0926 15:16:04.056783 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:04.056799 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:04.056915 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:04.057192 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:04.057195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:04.057420 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:04.057548 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:04.057554 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:04.058300 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:04.058303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:04.059123 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:04.059133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:04.059727 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:04.059731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:04.059847 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:04.060511 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:04.060527 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:04.063494 (+ 2967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":81}
W20260926 15:16:04.063910 16195 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:04.056774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:50973 (local address 127.6.209.254:44707)
0926 15:16:04.056980 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:04.056984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:04.056997 (+ 13us) server_negotiation.cc:407] Connection header received
0926 15:16:04.057027 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:04.057030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:04.057081 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:04.057151 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:04.057637 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:04.058205 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:04.059227 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:04.059617 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:04.059700 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:04.063666 (+ 3966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.195:50973: BlockingRecv error: recv got EOF from 127.6.209.195:50973 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
W20260926 15:16:04.064033 16192 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 15:16:04 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:04 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:04.593379 16199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:04.585933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.193:36159)
0926 15:16:04.586402 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:04.586417 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:04.586505 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:04.586776 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:04.586778 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:04.587124 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:04.587242 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:04.587249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:04.588055 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:04.588058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:04.588967 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:04.588978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:04.589594 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:04.589598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:04.589718 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:04.590257 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:04.590272 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:04.593153 (+ 2881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":254,"threads_started":1}
W20260926 15:16:04.593549 16200 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:04.586015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:36159 (local address 127.6.209.254:44707)
0926 15:16:04.586449 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:04.586452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:04.586466 (+ 14us) server_negotiation.cc:407] Connection header received
0926 15:16:04.586606 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:04.586609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:04.586661 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:04.586740 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:04.587341 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:04.587946 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:04.589080 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:04.589456 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:04.589543 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:04.593348 (+ 3805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.193:36159: BlockingRecv error: recv got EOF from 127.6.209.193:36159 (error 108)
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":245,"threads_started":1}
Sep 26 15:16:04 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:04 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:04.923295 16201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:04.910294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.194:35585)
0926 15:16:04.910739 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:04.910752 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:04.910888 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:04.911181 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:04.911185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:04.911541 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:04.911672 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:04.911679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:04.912546 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:04.912552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:04.913384 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:04.913393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:04.914022 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:04.914028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:04.914183 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:04.919975 (+ 5792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:04.919991 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:04.923041 (+ 3050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":252,"threads_started":1}
W20260926 15:16:04.923476 16200 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:04.910343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:35585 (local address 127.6.209.254:44707)
0926 15:16:04.910527 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:04.910534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:04.910829 (+ 295us) server_negotiation.cc:407] Connection header received
0926 15:16:04.910993 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:04.910998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:04.911054 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:04.911147 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:04.911796 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:04.912420 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:04.913503 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:04.913863 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:04.913990 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:04.923206 (+ 9216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.194:35585: BlockingRecv error: recv got EOF from 127.6.209.194:35585 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 15:16:05 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:05 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:05.072531 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:05.064483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.195:60373)
0926 15:16:05.064977 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:05.064990 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:05.065132 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:05.065452 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:05.065457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:05.065842 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:05.065968 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:05.065975 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:05.066853 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:05.066859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:05.067656 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:05.067665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:05.068339 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:05.068345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:05.068485 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:05.069119 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:05.069137 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:05.072321 (+ 3184us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":318,"threads_started":1}
W20260926 15:16:05.072690 16200 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:05.064544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:60373 (local address 127.6.209.254:44707)
0926 15:16:05.064735 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:05.064742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:05.065073 (+ 331us) server_negotiation.cc:407] Connection header received
0926 15:16:05.065270 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:05.065274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:05.065333 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:05.065426 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:05.066094 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:05.066737 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:05.067799 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:05.068188 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:05.068309 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:05.072479 (+ 4170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.195:60373: BlockingRecv error: recv got EOF from 127.6.209.195:60373 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 15:16:05 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:05 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:05.605559 16203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:05.594219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.193:56511)
0926 15:16:05.594690 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:05.594720 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:05.594812 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:05.595070 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:05.595073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:05.595383 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:05.595532 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:05.595539 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:05.596322 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:05.596325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:05.597155 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:05.597167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:05.597769 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:05.597773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:05.597906 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:05.602216 (+ 4310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:05.602231 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:05.605346 (+ 3115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":258,"threads_started":1}
W20260926 15:16:05.605798 16204 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:05.594295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:56511 (local address 127.6.209.254:44707)
0926 15:16:05.594695 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:05.594715 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:05.594781 (+ 66us) server_negotiation.cc:407] Connection header received
0926 15:16:05.594895 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:05.594898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:05.594956 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:05.595032 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:05.595640 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:05.596226 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:05.597273 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:05.597652 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:05.597740 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:05.605547 (+ 7807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.193:56511: BlockingRecv error: recv got EOF from 127.6.209.193:56511 (error 108)
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":212,"threads_started":1}
Sep 26 15:16:05 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:05 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:05.931784 16205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:05.924191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.194:39783)
0926 15:16:05.924590 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:05.924612 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:05.924714 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:05.924986 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:05.924989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:05.925321 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:05.925484 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:05.925491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:05.926340 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:05.926343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:05.927254 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:05.927264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:05.927886 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:05.927889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:05.928008 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:05.928531 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:05.928546 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:05.931572 (+ 3026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":213,"threads_started":1}
W20260926 15:16:05.931879 16204 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:05.924195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:39783 (local address 127.6.209.254:44707)
0926 15:16:05.924379 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:05.924388 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:05.924691 (+ 303us) server_negotiation.cc:407] Connection header received
0926 15:16:05.924800 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:05.924819 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:05.924876 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:05.924947 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:05.925599 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:05.926242 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:05.927381 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:05.927770 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:05.927858 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:05.931722 (+ 3864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.194:39783: BlockingRecv error: recv got EOF from 127.6.209.194:39783 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 15:16:06 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:06 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:06.081511 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:06.073353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.195:56109)
0926 15:16:06.073869 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:06.073883 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:06.074052 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:06.074394 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:06.074398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:06.074792 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:06.074961 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:06.074969 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:06.075951 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:06.075957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:06.076880 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:06.076890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:06.077485 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:06.077489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:06.077654 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:06.078186 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:06.078201 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:06.081301 (+ 3100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":312,"threads_started":1}
W20260926 15:16:06.081763 16204 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:06.073472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:56109 (local address 127.6.209.254:44707)
0926 15:16:06.073627 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:06.073631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:06.073981 (+ 350us) server_negotiation.cc:407] Connection header received
0926 15:16:06.074174 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:06.074179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:06.074258 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:06.074366 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:06.075108 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:06.075817 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:06.077003 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:06.077357 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:06.077473 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:06.081455 (+ 3982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.195:56109: BlockingRecv error: recv got EOF from 127.6.209.195:56109 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 15:16:06 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:06 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:06.614598 16207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:06.606599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.193:37131)
0926 15:16:06.606973 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:06.606987 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:06.607074 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:06.607468 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:06.607470 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:06.607765 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:06.607920 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:06.607927 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:06.608739 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:06.608742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:06.609695 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:06.609705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:06.610308 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:06.610311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:06.610467 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:06.611068 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:06.611084 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:06.614308 (+ 3224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":199,"threads_started":1}
W20260926 15:16:06.614675 16208 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:06.606667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:37131 (local address 127.6.209.254:44707)
0926 15:16:06.607250 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:06.607254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:06.607266 (+ 12us) server_negotiation.cc:407] Connection header received
0926 15:16:06.607301 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:06.607304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:06.607358 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:06.607432 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:06.608034 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:06.608637 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:06.609808 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:06.610188 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:06.610285 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:06.614462 (+ 4177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.193:37131: BlockingRecv error: recv got EOF from 127.6.209.193:37131 (error 108)
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":423,"threads_started":1}
Sep 26 15:16:06 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:06 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:06.941473 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:06.932822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.194:39013)
0926 15:16:06.933303 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:06.933317 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:06.933412 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:06.933676 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:06.933678 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:06.934024 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:06.934188 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:06.934196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:06.935104 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:06.935107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:06.935931 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:06.935939 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:06.936691 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:06.936698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:06.936912 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:06.937583 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:06.937606 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:06.941233 (+ 3627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":248,"threads_started":1}
W20260926 15:16:06.941576 16208 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:06.932903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:39013 (local address 127.6.209.254:44707)
0926 15:16:06.933064 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:06.933068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:06.933383 (+ 315us) server_negotiation.cc:407] Connection header received
0926 15:16:06.933502 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:06.933505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:06.933562 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:06.933643 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:06.934304 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:06.934989 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:06.936052 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:06.936537 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:06.936679 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:06.941378 (+ 4699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.194:39013: BlockingRecv error: recv got EOF from 127.6.209.194:39013 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 15:16:07 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:07 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:07.090063 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:07.082470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.195:55659)
0926 15:16:07.082919 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:07.082932 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:07.083027 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:07.083273 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:07.083275 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:07.083563 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:07.083707 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:07.083713 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:07.084607 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:07.084611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:07.085398 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:07.085407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:07.085939 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:07.085942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:07.086040 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:07.086650 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:07.086664 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:07.089849 (+ 3185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":238,"threads_started":1}
W20260926 15:16:07.090210 16208 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:07.082610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:55659 (local address 127.6.209.254:44707)
0926 15:16:07.082777 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:07.082781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:07.082995 (+ 214us) server_negotiation.cc:407] Connection header received
0926 15:16:07.083106 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:07.083109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:07.083167 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:07.083237 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:07.083809 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:07.084379 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:07.085497 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:07.085839 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:07.085917 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:07.090005 (+ 4088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.195:55659: BlockingRecv error: recv got EOF from 127.6.209.195:55659 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 15:16:07 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:07 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:07.623582 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:07.615487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.193:38401)
0926 15:16:07.615901 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:07.615919 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:07.616029 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:07.616451 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:07.616455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:07.616994 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:07.617214 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:07.617228 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:07.618089 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:07.618092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:07.618909 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:07.618917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:07.619540 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:07.619543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:07.619662 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:07.620183 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:07.620200 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:07.623255 (+ 3055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":204,"threads_started":1}
W20260926 15:16:07.623687 16212 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:07.615548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:38401 (local address 127.6.209.254:44707)
0926 15:16:07.616157 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:07.616164 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:07.616185 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:16:07.616246 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:07.616248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:07.616320 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:07.616394 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:07.617349 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:07.617991 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:07.619022 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:07.619419 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:07.619504 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:07.623412 (+ 3908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.193:38401: BlockingRecv error: recv got EOF from 127.6.209.193:38401 (error 108)
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":424,"threads_started":1}
Sep 26 15:16:07 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:07 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:07.951058 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:07.942327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.194:33891)
0926 15:16:07.942815 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:07.942831 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:07.942932 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:07.943210 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:07.943213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:07.943558 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:07.943714 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:07.943722 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:07.944747 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:07.944751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:07.945873 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:07.945884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:07.946479 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:07.946483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:07.946603 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:07.947249 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:07.947266 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:07.950828 (+ 3562us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":280,"threads_started":1}
W20260926 15:16:07.951182 16212 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:07.942425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:33891 (local address 127.6.209.254:44707)
0926 15:16:07.942581 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:07.942586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:07.942899 (+ 313us) server_negotiation.cc:407] Connection header received
0926 15:16:07.943023 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:07.943026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:07.943084 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:07.943165 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:07.943825 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:07.944627 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:07.945988 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:07.946367 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:07.946468 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:07.950987 (+ 4519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.194:33891: BlockingRecv error: recv got EOF from 127.6.209.194:33891 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 15:16:08 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:08 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:08.098428 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:08.090868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.195:46437)
0926 15:16:08.091331 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:08.091352 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:08.091445 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:08.091693 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:08.091696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:08.092001 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:08.092135 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:08.092142 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:08.093022 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:08.093028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:08.094019 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:08.094031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:08.094642 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:08.094646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:08.094758 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:08.095247 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:08.095262 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:08.098193 (+ 2931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":264,"threads_started":1}
W20260926 15:16:08.098585 16212 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:08.090984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:46437 (local address 127.6.209.254:44707)
0926 15:16:08.091168 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:08.091172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:08.091411 (+ 239us) server_negotiation.cc:407] Connection header received
0926 15:16:08.091521 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:08.091524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:08.091584 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:08.091653 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:08.092230 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:08.092901 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:08.094138 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:08.094526 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:08.094614 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:08.098350 (+ 3736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.195:46437: BlockingRecv error: recv got EOF from 127.6.209.195:46437 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 15:16:08 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:08 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:08.632639 16215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:08.624384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.193:59699)
0926 15:16:08.625119 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:08.625132 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:08.625276 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:08.625564 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:08.625568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:08.626045 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:08.626188 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:08.626196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:08.627099 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:08.627104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:08.627960 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:08.627971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:08.628679 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:08.628684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:08.628871 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:08.629445 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:08.629462 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:08.632413 (+ 2951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":565,"threads_started":1}
W20260926 15:16:08.632822 16216 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:08.624409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:59699 (local address 127.6.209.254:44707)
0926 15:16:08.624897 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:08.624901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:08.625209 (+ 308us) server_negotiation.cc:407] Connection header received
0926 15:16:08.625386 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:08.625391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:08.625446 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:08.625538 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:08.626316 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:08.626983 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:08.628082 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:08.628555 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:08.628660 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:08.632594 (+ 3934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.193:59699: BlockingRecv error: recv got EOF from 127.6.209.193:59699 (error 108)
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":284,"threads_started":1}
Sep 26 15:16:08 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:08 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:08.960028 16217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:08.951905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.194:44217)
0926 15:16:08.952354 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:08.952368 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:08.952498 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:08.952852 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:08.952857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:08.953276 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:08.953425 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:08.953433 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:08.954312 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:08.954317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:08.955210 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:08.955221 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:08.955854 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:08.955860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:08.956039 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:08.956670 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:08.956685 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:08.959722 (+ 3037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":232,"threads_started":1}
W20260926 15:16:08.960114 16216 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:08.952094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:44217 (local address 127.6.209.254:44707)
0926 15:16:08.952286 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:08.952292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:08.952437 (+ 145us) server_negotiation.cc:407] Connection header received
0926 15:16:08.952646 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:08.952651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:08.952711 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:08.952802 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:08.953545 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:08.954190 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:08.955337 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:08.955717 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:08.955832 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:08.959896 (+ 4064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.194:44217: BlockingRecv error: recv got EOF from 127.6.209.194:44217 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 15:16:09 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:09 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:09.107529 16218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:09.099238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.195:37595)
0926 15:16:09.099790 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:09.099805 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:09.099952 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:09.100273 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:09.100277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:09.100900 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:09.101056 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:09.101064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:09.102178 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:09.102183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:09.103085 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:09.103094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:09.103701 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:09.103707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:09.103867 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:09.104340 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:09.104356 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:09.107322 (+ 2966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":312,"threads_started":1}
W20260926 15:16:09.107714 16216 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:09.099376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:37595 (local address 127.6.209.254:44707)
0926 15:16:09.099626 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:09.099633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:09.099889 (+ 256us) server_negotiation.cc:407] Connection header received
0926 15:16:09.100070 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:09.100086 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:09.100151 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:09.100252 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:09.101188 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:09.102042 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:09.103209 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:09.103575 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:09.103690 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:09.107492 (+ 3802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.195:37595: BlockingRecv error: recv got EOF from 127.6.209.195:37595 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 26 15:16:09 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:09 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:09.641212 16219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:09.633567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.193:58285)
0926 15:16:09.634052 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:09.634066 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:09.634172 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:09.634438 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:09.634441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:09.634777 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:09.634913 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:09.634921 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:09.635762 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:09.635765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:09.636570 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:09.636580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:09.637193 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:09.637197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:09.637304 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:09.637785 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:09.637800 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:09.640998 (+ 3198us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":273,"threads_started":1}
W20260926 15:16:09.641490 16220 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:09.633611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:58285 (local address 127.6.209.254:44707)
0926 15:16:09.634066 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:09.634070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:09.634120 (+ 50us) server_negotiation.cc:407] Connection header received
0926 15:16:09.634270 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:09.634274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:09.634325 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:09.634400 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:09.635016 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:09.635651 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:09.636681 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:09.637082 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:09.637161 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:09.641193 (+ 4032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.193:58285: BlockingRecv error: recv got EOF from 127.6.209.193:58285 (error 108)
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":251,"threads_started":1}
Sep 26 15:16:09 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:09 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:09.972772 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:09.960937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.194:41053)
0926 15:16:09.961380 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:09.961394 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:09.961540 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:09.961878 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:09.961882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:09.962270 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:09.962407 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:09.962413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:09.963356 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:09.963362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:09.964182 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:09.964192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:09.964860 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:09.964865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:09.965037 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:09.969398 (+ 4361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:09.969415 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:09.972453 (+ 3038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":247,"threads_started":1}
W20260926 15:16:09.972815 16220 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:09.961000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:41053 (local address 127.6.209.254:44707)
0926 15:16:09.961207 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:09.961216 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:09.961473 (+ 257us) server_negotiation.cc:407] Connection header received
0926 15:16:09.961675 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:09.961679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:09.961733 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:09.961840 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:09.962526 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:09.963234 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:09.964306 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:09.964695 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:09.964815 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:09.972625 (+ 7810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.194:41053: BlockingRecv error: recv got EOF from 127.6.209.194:41053 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 15:16:10 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:10 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:10.117281 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:10.108412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.195:49343)
0926 15:16:10.108931 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:10.108946 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:10.109095 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:10.109412 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:10.109417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:10.109831 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:10.109981 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:10.109989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:10.111122 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:10.111127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:10.112034 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:10.112044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:10.112728 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:10.112734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:10.112934 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:10.113476 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:10.113493 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:10.116954 (+ 3461us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":217,"threads_started":1}
W20260926 15:16:10.117385 16220 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:10.108492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:49343 (local address 127.6.209.254:44707)
0926 15:16:10.108745 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:10.108751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:10.109023 (+ 272us) server_negotiation.cc:407] Connection header received
0926 15:16:10.109212 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:10.109217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:10.109291 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:10.109390 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:10.110121 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:10.110989 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:10.112178 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:10.112571 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:10.112723 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:10.117108 (+ 4385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.195:49343: BlockingRecv error: recv got EOF from 127.6.209.195:49343 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 15:16:10 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:10 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:10.652427 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:10.642069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.193:45691)
0926 15:16:10.642566 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:10.642586 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:10.642700 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:10.643100 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:10.643106 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:10.643469 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:10.643617 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:10.643625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:10.644921 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:10.644927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:10.646505 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:10.646536 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:10.647923 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:10.647932 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:10.648116 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:10.648791 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:10.648835 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:10.652208 (+ 3373us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":220,"threads_started":1}
W20260926 15:16:10.652598 16224 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:10.642198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:45691 (local address 127.6.209.254:44707)
0926 15:16:10.642756 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:10.642763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:10.642777 (+ 14us) server_negotiation.cc:407] Connection header received
0926 15:16:10.642820 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:10.642824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:10.642935 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:10.643054 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:10.643745 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:10.644763 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:10.646671 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:10.647598 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:10.647702 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:10.652351 (+ 4649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.193:45691: BlockingRecv error: recv got EOF from 127.6.209.193:45691 (error 108)
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":382,"threads_started":1}
Sep 26 15:16:10 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:10 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:10.981894 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:10.973813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.194:35959)
0926 15:16:10.974395 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:10.974410 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:10.974513 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:10.974786 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:10.974789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:10.975175 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:10.975329 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:10.975336 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:10.976222 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:10.976225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:10.977134 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:10.977144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:10.977706 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:10.977710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:10.977809 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:10.978318 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:10.978333 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:10.981577 (+ 3244us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":279,"threads_started":1}
W20260926 15:16:10.981942 16224 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:10.973846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:35959 (local address 127.6.209.254:44707)
0926 15:16:10.974011 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:10.974015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:10.974477 (+ 462us) server_negotiation.cc:407] Connection header received
0926 15:16:10.974611 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:10.974615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:10.974671 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:10.974745 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:10.975439 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:10.976115 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:10.977243 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:10.977593 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:10.977678 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:10.981733 (+ 4055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.194:35959: BlockingRecv error: recv got EOF from 127.6.209.194:35959 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 15:16:11 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:11 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:11.127383 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:11.118326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.195:43045)
0926 15:16:11.118835 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:11.118860 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:11.118969 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:11.119225 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:11.119228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:11.119612 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:11.119786 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:11.119795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:11.120757 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:11.120761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:11.122164 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:11.122176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:11.122969 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:11.122975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:11.123141 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:11.123763 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:11.123780 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:11.127027 (+ 3247us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":251,"threads_started":1}
W20260926 15:16:11.127457 16224 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:11.118352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:43045 (local address 127.6.209.254:44707)
0926 15:16:11.118522 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:11.118525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:11.118929 (+ 404us) server_negotiation.cc:407] Connection header received
0926 15:16:11.119054 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:11.119057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:11.119114 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:11.119181 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:11.119911 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:11.120604 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:11.122311 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:11.122762 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:11.122962 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:11.127223 (+ 4261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.195:43045: BlockingRecv error: recv got EOF from 127.6.209.195:43045 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 15:16:11 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:11 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:11.661415 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:11.653353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.193:55021)
0926 15:16:11.653834 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:11.653853 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:11.653978 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:11.654288 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:11.654291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:11.654614 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:11.654752 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:11.654759 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:11.655648 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:11.655651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:11.656535 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:11.656556 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:11.657256 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:11.657260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:11.657360 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:11.657825 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:11.657840 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:11.661074 (+ 3234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":219,"threads_started":1}
W20260926 15:16:11.661507 16228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:11.653536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:55021 (local address 127.6.209.254:44707)
0926 15:16:11.653963 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:11.653968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:11.653983 (+ 15us) server_negotiation.cc:407] Connection header received
0926 15:16:11.654122 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:11.654125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:11.654175 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:11.654248 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:11.654850 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:11.655530 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:11.656658 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:11.657126 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:11.657205 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:11.661253 (+ 4048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.193:55021: BlockingRecv error: recv got EOF from 127.6.209.193:55021 (error 108)
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":223,"threads_started":1}
Sep 26 15:16:11 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:11 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 15:16:11.991766 16228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:11.982965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:51547 (local address 127.6.209.254:44707)
0926 15:16:11.983176 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:11.983183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:11.983411 (+ 228us) server_negotiation.cc:407] Connection header received
0926 15:16:11.983520 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:11.983524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:11.983584 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:11.983664 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:11.984305 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:11.985341 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:11.986524 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:11.986932 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:11.987072 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:11.991615 (+ 4543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.194:51547: BlockingRecv error: recv got EOF from 127.6.209.194:51547 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 15:16:11.991771 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:11.982876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.194:51547)
0926 15:16:11.983313 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:11.983328 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:11.983430 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:11.983697 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:11.983700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:11.984053 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:11.984193 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:11.984200 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:11.985485 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:11.985491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:11.986378 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:11.986390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:11.987106 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:11.987113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:11.987283 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:11.987862 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:11.987878 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:11.991477 (+ 3599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":217,"threads_started":1}
Sep 26 15:16:12 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:12 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:12.137056 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:12.128380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.195:45929)
0926 15:16:12.128885 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:12.128898 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:12.129029 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:12.129318 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:12.129322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:12.129714 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:12.129866 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:12.129874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:12.130836 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:12.130841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:12.131728 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:12.131737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:12.132402 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:12.132407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:12.132553 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:12.133121 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:12.133137 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:12.136694 (+ 3557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":251,"threads_started":1}
W20260926 15:16:12.137140 16228 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:12.128484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:45929 (local address 127.6.209.254:44707)
0926 15:16:12.128686 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:12.128693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:12.128977 (+ 284us) server_negotiation.cc:407] Connection header received
0926 15:16:12.129156 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:12.129160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:12.129209 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:12.129293 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:12.130004 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:12.130707 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:12.131856 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:12.132242 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:12.132354 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:12.136887 (+ 4533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.195:45929: BlockingRecv error: recv got EOF from 127.6.209.195:45929 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 15:16:12 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:12 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:12.670511 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:12.662358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.193:34263)
0926 15:16:12.662792 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:12.662806 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:12.662898 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:12.663463 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:12.663467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:12.663818 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:12.663952 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:12.663960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:12.664889 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:12.664892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:12.665976 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:12.665986 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:12.666766 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:12.666770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:12.666882 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:12.667345 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:12.667360 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:12.670285 (+ 2925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":198,"threads_started":1}
W20260926 15:16:12.670701 16231 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:12.662382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:34263 (local address 127.6.209.254:44707)
0926 15:16:12.663212 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:12.663216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:12.663227 (+ 11us) server_negotiation.cc:407] Connection header received
0926 15:16:12.663265 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:12.663268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:12.663315 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:12.663401 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:12.664063 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:12.664751 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:12.666101 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:12.666650 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:12.666736 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:12.670453 (+ 3717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.193:34263: BlockingRecv error: recv got EOF from 127.6.209.193:34263 (error 108)
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":641,"threads_started":1}
Sep 26 15:16:12 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:12 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:13.001108 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:12.992947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.194:39603)
0926 15:16:12.993345 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:12.993359 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:12.993459 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:12.993720 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:12.993723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:12.994084 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:12.994212 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:12.994219 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:12.995046 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:12.995054 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:12.995849 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:12.995858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:12.996539 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:12.996543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:12.996745 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:12.997439 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:12.997460 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:13.000763 (+ 3303us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":212,"threads_started":1}
W20260926 15:16:13.001184 16231 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:12.993014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:39603 (local address 127.6.209.254:44707)
0926 15:16:12.993219 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:12.993224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:12.993436 (+ 212us) server_negotiation.cc:407] Connection header received
0926 15:16:12.993546 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:12.993549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:12.993612 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:12.993681 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:12.994317 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:12.994956 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:12.995953 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:12.996419 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:12.996506 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:13.000956 (+ 4450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.194:39603: BlockingRecv error: recv got EOF from 127.6.209.194:39603 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 15:16:13 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:13 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:13.145627 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:13.137906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.195:57711)
0926 15:16:13.138320 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:13.138335 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:13.138430 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:13.138741 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:13.138745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:13.139093 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:13.139216 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:13.139222 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:13.140097 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:13.140101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:13.141064 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:13.141075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:13.141663 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:13.141667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:13.141792 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:13.142342 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:13.142357 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:13.145428 (+ 3071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":196,"threads_started":1}
W20260926 15:16:13.145795 16231 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:13.137889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:57711 (local address 127.6.209.254:44707)
0926 15:16:13.138071 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:13.138078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:13.138404 (+ 326us) server_negotiation.cc:407] Connection header received
0926 15:16:13.138515 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:13.138518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:13.138592 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:13.138662 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:13.139321 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:13.139995 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:13.141170 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:13.141545 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:13.141637 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:13.145571 (+ 3934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.195:57711: BlockingRecv error: recv got EOF from 127.6.209.195:57711 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 15:16:13 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:13 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:13.679703 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:13.671477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.193:34373)
0926 15:16:13.672013 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:13.672029 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:13.672153 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:13.672431 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:13.672434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:13.672750 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:13.672955 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:13.672963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:13.673880 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:13.673883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:13.674816 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:13.674825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:13.675493 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:13.675497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:13.675778 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:13.676398 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:13.676413 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:13.679490 (+ 3077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":274,"threads_started":1}
W20260926 15:16:13.680106 16236 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:13.671571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:34373 (local address 127.6.209.254:44707)
0926 15:16:13.672060 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:13.672065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:13.672097 (+ 32us) server_negotiation.cc:407] Connection header received
0926 15:16:13.672250 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:13.672253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:13.672309 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:13.672392 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:13.673091 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:13.673776 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:13.674920 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:13.675329 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:13.675463 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:13.679673 (+ 4210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.193:34373: BlockingRecv error: recv got EOF from 127.6.209.193:34373 (error 108)
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":265,"threads_started":1}
Sep 26 15:16:14 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:14 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:14.009433 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:14.001939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.194:33521)
0926 15:16:14.002399 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:14.002413 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:14.002506 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:14.002751 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:14.002754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:14.003033 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:14.003181 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:14.003187 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:14.004070 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:14.004073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:14.004949 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:14.004958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:14.005476 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:14.005480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:14.005569 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:14.006071 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:14.006085 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:14.009130 (+ 3045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":270,"threads_started":1}
W20260926 15:16:14.009482 16236 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:14.002021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:33521 (local address 127.6.209.254:44707)
0926 15:16:14.002163 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:14.002167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:14.002475 (+ 308us) server_negotiation.cc:407] Connection header received
0926 15:16:14.002590 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:14.002593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:14.002644 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:14.002710 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:14.003282 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:14.003961 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:14.005048 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:14.005373 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:14.005448 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:14.009287 (+ 3839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.194:33521: BlockingRecv error: recv got EOF from 127.6.209.194:33521 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 15:16:14 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:14 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:14.154608 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:14.146419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.195:49859)
0926 15:16:14.146867 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:14.146881 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:14.147011 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:14.147337 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:14.147341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:14.147816 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:14.148025 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:14.148035 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:14.149207 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:14.149213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:14.150148 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:14.150157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:14.150809 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:14.150814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:14.150976 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:14.151516 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:14.151531 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:14.154307 (+ 2776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":265,"threads_started":1}
W20260926 15:16:14.154680 16236 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:14.146573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:49859 (local address 127.6.209.254:44707)
0926 15:16:14.146762 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:14.146768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:14.146948 (+ 180us) server_negotiation.cc:407] Connection header received
0926 15:16:14.147121 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:14.147126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:14.147196 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:14.147314 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:14.148178 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:14.149084 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:14.150282 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:14.150667 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:14.150799 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:14.154450 (+ 3651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.195:49859: BlockingRecv error: recv got EOF from 127.6.209.195:49859 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 15:16:14 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:14 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:14.689574 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:14.680561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.193:38867)
0926 15:16:14.681077 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:14.681096 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:14.681239 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:14.681657 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:14.681661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:14.682134 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:14.682373 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:14.682382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:14.683196 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:14.683200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:14.684456 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:14.684467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:14.685213 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:14.685219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:14.685384 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:14.685893 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:14.685912 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:14.689332 (+ 3420us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":265,"threads_started":1}
W20260926 15:16:14.689666 16240 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:14.680622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:38867 (local address 127.6.209.254:44707)
0926 15:16:14.681157 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:14.681175 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:14.681191 (+ 16us) server_negotiation.cc:407] Connection header received
0926 15:16:14.681361 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:14.681403 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:14.681502 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:14.681605 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:14.682490 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:14.683098 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:14.684573 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:14.685088 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:14.685185 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:14.689479 (+ 4294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.193:38867: BlockingRecv error: recv got EOF from 127.6.209.193:38867 (error 108)
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":308,"threads_started":1}
Sep 26 15:16:15 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:15 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:15.023880 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:15.010420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.194:36289)
0926 15:16:15.010842 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:15.010858 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:15.010965 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:15.011245 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:15.011248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:15.011591 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:15.011746 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:15.011753 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:15.012611 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:15.012614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:15.013472 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:15.013482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:15.014030 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:15.014034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:15.014156 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:15.020089 (+ 5933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:15.020106 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:15.023581 (+ 3475us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":235,"threads_started":1}
W20260926 15:16:15.023919 16240 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:15.010460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:36289 (local address 127.6.209.254:44707)
0926 15:16:15.010663 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:15.010672 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:15.010934 (+ 262us) server_negotiation.cc:407] Connection header received
0926 15:16:15.011048 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:15.011051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:15.011108 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:15.011233 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:15.011848 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:15.012513 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:15.013576 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:15.013922 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:15.014002 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:15.023733 (+ 9731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.194:36289: BlockingRecv error: recv got EOF from 127.6.209.194:36289 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 15:16:15 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:15 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:15.163091 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:15.155562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.195:50405)
0926 15:16:15.155966 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:15.155980 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:15.156075 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:15.156322 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:15.156325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:15.156660 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:15.156841 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:15.156849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:15.157665 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:15.157671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:15.158429 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:15.158439 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:15.159041 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:15.159047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:15.159183 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:15.159657 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:15.159672 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:15.162795 (+ 3123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":227,"threads_started":1}
W20260926 15:16:15.163203 16240 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:15.155649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:50405 (local address 127.6.209.254:44707)
0926 15:16:15.155786 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:15.155790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:15.156051 (+ 261us) server_negotiation.cc:407] Connection header received
0926 15:16:15.156158 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:15.156161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:15.156220 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:15.156289 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:15.156944 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:15.157542 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:15.158552 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:15.158894 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:15.159003 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:15.162945 (+ 3942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.195:50405: BlockingRecv error: recv got EOF from 127.6.209.195:50405 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 15:16:15 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:15 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:15.698269 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:15.690404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.193:56897)
0926 15:16:15.690874 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:15.690889 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:15.690992 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:15.691299 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:15.691302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:15.691650 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:15.691819 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:15.691826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:15.692805 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:15.692808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:15.693632 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:15.693641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:15.694212 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:15.694216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:15.694341 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:15.694841 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:15.694855 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:15.697926 (+ 3071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":244,"threads_started":1}
W20260926 15:16:15.698381 16244 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:15.690445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:56897 (local address 127.6.209.254:44707)
0926 15:16:15.690960 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:15.690964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:15.690983 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:16:15.691094 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:15.691097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:15.691169 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:15.691258 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:15.691961 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:15.692680 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:15.693742 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:15.694110 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:15.694189 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:15.698121 (+ 3932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.193:56897: BlockingRecv error: recv got EOF from 127.6.209.193:56897 (error 108)
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":269,"threads_started":1}
Sep 26 15:16:16 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:16 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:16.032320 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:16.024650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.194:46655)
0926 15:16:16.025170 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:16.025185 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:16.025278 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:16.025512 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:16.025515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:16.025846 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:16.026007 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:16.026014 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:16.026843 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:16.026846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:16.027697 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:16.027706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:16.028268 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:16.028271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:16.028373 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:16.028902 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:16.028917 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:16.032004 (+ 3087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":290,"threads_started":1}
W20260926 15:16:16.032377 16244 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:16.024677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:46655 (local address 127.6.209.254:44707)
0926 15:16:16.024864 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:16.024869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:16.025244 (+ 375us) server_negotiation.cc:407] Connection header received
0926 15:16:16.025356 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:16.025359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:16.025409 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:16.025477 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:16.026114 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:16.026743 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:16.027793 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:16.028144 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:16.028217 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:16.032169 (+ 3952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.194:46655: BlockingRecv error: recv got EOF from 127.6.209.194:46655 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 15:16:16 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:16 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:16.171836 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:16.163932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.195:44901)
0926 15:16:16.164448 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:16.164462 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:16.164558 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:16.164861 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:16.164866 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:16.165235 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:16.165374 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:16.165382 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:16.166242 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:16.166247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:16.167066 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:16.167075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:16.167725 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:16.167731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:16.167886 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:16.168435 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:16.168451 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:16.171547 (+ 3096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":304,"threads_started":1}
W20260926 15:16:16.171926 16244 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:16.164022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:44901 (local address 127.6.209.254:44707)
0926 15:16:16.164188 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:16.164191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:16.164523 (+ 332us) server_negotiation.cc:407] Connection header received
0926 15:16:16.164652 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:16.164658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:16.164720 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:16.164813 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:16.165505 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:16.166121 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:16.167183 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:16.167539 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:16.167684 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:16.171702 (+ 4018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.195:44901: BlockingRecv error: recv got EOF from 127.6.209.195:44901 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 15:16:16 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:16 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:16.706923 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:16.699030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.193:46113)
0926 15:16:16.699429 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:16.699453 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:16.699563 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:16.699974 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:16.699977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:16.700328 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:16.700463 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:16.700471 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:16.701455 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:16.701458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:16.702334 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:16.702344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:16.702952 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:16.702956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:16.703062 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:16.703585 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:16.703601 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:16.706740 (+ 3139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":207,"threads_started":1}
W20260926 15:16:16.707123 16248 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:16.699252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:46113 (local address 127.6.209.254:44707)
0926 15:16:16.699753 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:16.699757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:16.699769 (+ 12us) server_negotiation.cc:407] Connection header received
0926 15:16:16.699804 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:16.699807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:16.699863 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:16.699936 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:16.700583 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:16.701312 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:16.702441 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:16.702851 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:16.702929 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:16.706894 (+ 3965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.193:46113: BlockingRecv error: recv got EOF from 127.6.209.193:46113 (error 108)
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":313,"threads_started":1}
Sep 26 15:16:17 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:17 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.194@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:17.041823 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:17.033334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.194:47225)
0926 15:16:17.033836 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:17.033850 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:17.033958 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:17.034206 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:17.034209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:17.034552 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:17.034676 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:17.034683 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:17.035533 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:17.035536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:17.036275 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:17.036283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:17.036906 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:17.036910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:17.037010 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:17.037552 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:17.037566 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:17.041494 (+ 3928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":286,"threads_started":1}
W20260926 15:16:17.041862 16248 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:17.033489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:47225 (local address 127.6.209.254:44707)
0926 15:16:17.033645 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:17.033650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:17.033925 (+ 275us) server_negotiation.cc:407] Connection header received
0926 15:16:17.034045 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:17.034047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:17.034096 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:17.034163 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:17.034774 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:17.035425 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:17.036389 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:17.036759 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:17.036884 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:17.041653 (+ 4769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.194:47225: BlockingRecv error: recv got EOF from 127.6.209.194:47225 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 15:16:17 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:17 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.195@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:17.181046 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:17.172781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.195:57101)
0926 15:16:17.173262 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:17.173281 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:17.173410 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:17.173768 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:17.173772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:17.174169 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:17.174293 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:17.174300 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:17.175361 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:17.175366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:17.176248 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:17.176258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:17.177064 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:17.177070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:17.177218 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:17.177793 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:17.177808 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:17.180767 (+ 2959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":208,"threads_started":1}
W20260926 15:16:17.181237 16248 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:17.172867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:57101 (local address 127.6.209.254:44707)
0926 15:16:17.173063 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:17.173070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:17.173343 (+ 273us) server_negotiation.cc:407] Connection header received
0926 15:16:17.173543 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:17.173548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:17.173632 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:17.173740 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:17.174415 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:17.175243 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:17.176388 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:17.176912 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:17.177035 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:17.180984 (+ 3949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.195:57101: BlockingRecv error: recv got EOF from 127.6.209.195:57101 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 15:16:17 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:16:17 dist-test-slave-j6q7 krb5kdc[15192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.209.193@KRBTEST.COM for oryx/127.6.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:16:17.716799 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:17.708014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44707 (local address 127.6.209.193:46751)
0926 15:16:17.708464 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:17.708482 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:17.708609 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:17.708939 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:17.708943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:17.709278 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:17.709438 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:17.709445 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:17.710581 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:17.710588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:17.711935 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:17.711946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:17.712605 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:17.712610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:17.712777 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:17.713334 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:17.713367 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:17.716467 (+ 3100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.209.254:44707: Server oryx/127.6.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":187,"threads_started":1}
W20260926 15:16:17.716897 16252 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:17.708099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:46751 (local address 127.6.209.254:44707)
0926 15:16:17.708565 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:17.708569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:17.708583 (+ 14us) server_negotiation.cc:407] Connection header received
0926 15:16:17.708728 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:17.708731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:17.708783 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:17.708911 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:17.709567 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:17.710437 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:17.712057 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:17.712473 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:17.712595 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:17.716639 (+ 4044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.193:46751: BlockingRecv error: recv got EOF from 127.6.209.193:46751 (error 108)
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":216,"threads_started":1}
I20260926 15:16:18.008986 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 15779
I20260926 15:16:18.019833 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:18.010046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41986 (local address 127.6.209.193:38315)
0926 15:16:18.010670 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:18.010677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:18.010706 (+ 29us) server_negotiation.cc:407] Connection header received
0926 15:16:18.010760 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:18.010767 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:18.010999 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:18.011159 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:18.012451 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:18.013617 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:18.014659 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:18.015146 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:18.015233 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:18.017394 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:18.017401 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:18.017413 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:16:18.017443 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:18.019417 (+ 1974us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.209.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 15:16:18.019646 (+ 229us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41986: Request ticket server kudu/127.6.209.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":314,"threads_started":1}
W20260926 15:16:18.019941 16254 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41986: Request ticket server kudu/127.6.209.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 15:16:18.019927 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 15779: Not authorized: Client connection negotiation failed: client connection to 127.6.209.193:38315: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.209.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 15:16:18.020213 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 15779
I20260926 15:16:18.036000 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 15930
W20260926 15:16:18.051993 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 15930: Not authorized: Client connection negotiation failed: client connection to 127.6.209.194:34683: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.209.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 15:16:18.052080 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:18.036914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35914 (local address 127.6.209.194:34683)
0926 15:16:18.037405 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:18.037410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:18.037433 (+ 23us) server_negotiation.cc:407] Connection header received
0926 15:16:18.037480 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:18.037486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:18.037684 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:18.037822 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:18.039421 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:18.040365 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:18.041605 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:18.042107 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:18.042195 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:18.045035 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:18.045044 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:18.045058 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:16:18.045096 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:18.051428 (+ 6332us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.209.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 15:16:18.051785 (+ 357us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35914: Request ticket server kudu/127.6.209.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":239,"threads_started":1}
I20260926 15:16:18.052150 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 15930
I20260926 15:16:18.067130 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 16063
W20260926 15:16:18.067183 16252 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:18.043003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:33411 (local address 127.6.209.254:44707)
0926 15:16:18.043154 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:18.043159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:18.043638 (+ 479us) server_negotiation.cc:407] Connection header received
0926 15:16:18.043884 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:18.043888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:18.043971 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:18.044057 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:18.050481 (+ 6424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:18.051441 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:18.066991 (+ 15550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.209.194:33411: BlockingRecv error: recv got EOF from 127.6.209.194:33411 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 15:16:18.078223 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:18.068283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54996 (local address 127.6.209.195:46669)
0926 15:16:18.068896 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:18.068903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:18.068926 (+ 23us) server_negotiation.cc:407] Connection header received
0926 15:16:18.068996 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:18.069003 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:18.069221 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:18.069359 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:18.071061 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:18.071968 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:18.073042 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:18.073517 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:18.073605 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:18.075705 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:18.075712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:18.075724 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:16:18.075753 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:18.077753 (+ 2000us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.209.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 15:16:18.077975 (+ 222us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54996: Request ticket server kudu/127.6.209.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":309,"threads_started":1}
W20260926 15:16:18.078243 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 16063: Not authorized: Client connection negotiation failed: client connection to 127.6.209.195:46669: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.209.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 15:16:18.078326 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 16063
W20260926 15:16:18.078326 16257 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54996: Request ticket server kudu/127.6.209.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 15:16:18.093349 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 15710
I20260926 15:16:18.094393 15772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:16:18.197805 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 15710
2026-09-26T15:16:18Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20388 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T15:16:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:16:18Z Disabled control of system clock
I20260926 15:16:18.269729 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44969
--webserver_interface=127.6.209.254
--webserver_port=0
--builtin_ntp_servers=127.6.209.212:36361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:44969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 15:16:18.553989 16262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:16:18.554232 16262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:16:18.554302 16262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:16:18.560768 16262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:16:18.560901 16262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:16:18.560992 16262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:16:18.561098 16262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:16:18.561175 16262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:16:18.568976 16262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:36361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:44969
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:16:18.569998 16262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:16:18.571626 16262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:16:18.581650 16267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:18.581844 16268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:18.581844 16270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:16:18.583039 16262 server_base.cc:1034] running on GCE node
I20260926 15:16:18.583716 16262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:16:18.584929 16262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:16:18.586102 16262 hybrid_clock.cc:648] HybridClock initialized: now 1790435778586049 us; error 51 us; skew 500 ppm
I20260926 15:16:18.586388 16262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:16:18.588342 16262 webserver.cc:460] Webserver started at http://127.6.209.254:35233/ using document root <none> and password file <none>
I20260926 15:16:18.589015 16262 fs_manager.cc:359] Metadata directory not provided
I20260926 15:16:18.589118 16262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:16:18.589432 16262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:16:18.591930 16262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "670a385b4aaa4e5f886c615daa2b0ccc"
format_stamp: "Formatted at 2026-09-26 15:16:18 on dist-test-slave-j6q7"
server_key: "0c719f0390c2d6b900d346b3c46a62e5"
server_key_iv: "43261846a1a7b488d652ad587b923fe6"
server_key_version: "encryptionkey@0"
I20260926 15:16:18.592640 16262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "670a385b4aaa4e5f886c615daa2b0ccc"
format_stamp: "Formatted at 2026-09-26 15:16:18 on dist-test-slave-j6q7"
server_key: "0c719f0390c2d6b900d346b3c46a62e5"
server_key_iv: "43261846a1a7b488d652ad587b923fe6"
server_key_version: "encryptionkey@0"
I20260926 15:16:18.597139 16262 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 15:16:18.600097 16276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:18.601286 16262 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:16:18.601457 16262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "670a385b4aaa4e5f886c615daa2b0ccc"
format_stamp: "Formatted at 2026-09-26 15:16:18 on dist-test-slave-j6q7"
server_key: "0c719f0390c2d6b900d346b3c46a62e5"
server_key_iv: "43261846a1a7b488d652ad587b923fe6"
server_key_version: "encryptionkey@0"
I20260926 15:16:18.601676 16262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:16:18.611737 16262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:16:18.612388 16262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:16:18.612664 16262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:16:18.628228 16262 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:44969
I20260926 15:16:18.628245 16327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:44969 every 8 connection(s)
I20260926 15:16:18.629266 16262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:16:18.632289 16328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:16:18.635202 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 16262
I20260926 15:16:18.635389 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:16:18.636086 6983 external_mini_cluster.cc:1428] Setting key 265bb529bae8fc932af96c99ee4048cf
I20260926 15:16:18.645540 16328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 670a385b4aaa4e5f886c615daa2b0ccc: Bootstrap starting.
I20260926 15:16:18.648792 16328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 670a385b4aaa4e5f886c615daa2b0ccc: Neither blocks nor log segments found. Creating new log.
I20260926 15:16:18.649842 16328 log.cc:824] T 00000000000000000000000000000000 P 670a385b4aaa4e5f886c615daa2b0ccc: Log is configured to *not* fsync() on all Append() calls
I20260926 15:16:18.651451 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:18.637868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42018 (local address 127.6.209.254:44969)
0926 15:16:18.638658 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:18.638682 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:18.638714 (+ 32us) server_negotiation.cc:407] Connection header received
0926 15:16:18.641194 (+ 2480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:18.641220 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:18.641898 (+ 678us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:18.642300 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:18.643045 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:18.644882 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:18.646700 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:18.647736 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:18.647887 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:18.649188 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:18.649194 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:18.649205 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:16:18.649693 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:18.649757 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:16:18.649773 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:18.649892 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:18.650055 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 15:16:18.650353 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":296,"threads_started":1}
I20260926 15:16:18.652469 16328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 670a385b4aaa4e5f886c615daa2b0ccc: No bootstrap required, opened a new log
I20260926 15:16:18.659161 16328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 670a385b4aaa4e5f886c615daa2b0ccc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "670a385b4aaa4e5f886c615daa2b0ccc" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44969 } }
I20260926 15:16:18.659469 16328 raft_consensus.cc:374] T 00000000000000000000000000000000 P 670a385b4aaa4e5f886c615daa2b0ccc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:16:18.659564 16328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 670a385b4aaa4e5f886c615daa2b0ccc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 670a385b4aaa4e5f886c615daa2b0ccc, State: Initialized, Role: FOLLOWER
I20260926 15:16:18.660013 16328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 670a385b4aaa4e5f886c615daa2b0ccc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "670a385b4aaa4e5f886c615daa2b0ccc" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44969 } }
I20260926 15:16:18.660218 16328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 670a385b4aaa4e5f886c615daa2b0ccc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:16:18.660370 16328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 670a385b4aaa4e5f886c615daa2b0ccc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:16:18.660531 16328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 670a385b4aaa4e5f886c615daa2b0ccc [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:16:18.662647 16328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 670a385b4aaa4e5f886c615daa2b0ccc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "670a385b4aaa4e5f886c615daa2b0ccc" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44969 } }
I20260926 15:16:18.663065 16328 leader_election.cc:302] T 00000000000000000000000000000000 P 670a385b4aaa4e5f886c615daa2b0ccc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 670a385b4aaa4e5f886c615daa2b0ccc; no voters:
I20260926 15:16:18.663542 16328 leader_election.cc:288] T 00000000000000000000000000000000 P 670a385b4aaa4e5f886c615daa2b0ccc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:16:18.663618 16333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 670a385b4aaa4e5f886c615daa2b0ccc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:16:18.664399 16333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 670a385b4aaa4e5f886c615daa2b0ccc [term 1 LEADER]: Becoming Leader. State: Replica: 670a385b4aaa4e5f886c615daa2b0ccc, State: Running, Role: LEADER
I20260926 15:16:18.664929 16333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 670a385b4aaa4e5f886c615daa2b0ccc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "670a385b4aaa4e5f886c615daa2b0ccc" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44969 } }
I20260926 15:16:18.665546 16328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 670a385b4aaa4e5f886c615daa2b0ccc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:16:18.667104 16335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 670a385b4aaa4e5f886c615daa2b0ccc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 670a385b4aaa4e5f886c615daa2b0ccc. Latest consensus state: current_term: 1 leader_uuid: "670a385b4aaa4e5f886c615daa2b0ccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "670a385b4aaa4e5f886c615daa2b0ccc" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44969 } } }
I20260926 15:16:18.667450 16335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 670a385b4aaa4e5f886c615daa2b0ccc [sys.catalog]: This master's current role is: LEADER
I20260926 15:16:18.667348 16334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 670a385b4aaa4e5f886c615daa2b0ccc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "670a385b4aaa4e5f886c615daa2b0ccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "670a385b4aaa4e5f886c615daa2b0ccc" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44969 } } }
I20260926 15:16:18.667716 16334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 670a385b4aaa4e5f886c615daa2b0ccc [sys.catalog]: This master's current role is: LEADER
I20260926 15:16:18.669297 16338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:16:18.674185 16338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:16:18.681424 16338 catalog_manager.cc:1347] Generated new cluster ID: 59b9be7f3aab402eaff37ec0ad07cff8
I20260926 15:16:18.681560 16338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:16:18.707532 16338 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:16:18.708643 16338 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:16:18.714289 16338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 670a385b4aaa4e5f886c615daa2b0ccc: Generated new TSK 0
I20260926 15:16:18.714956 16338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:16:18.729951 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44969
--builtin_ntp_servers=127.6.209.212:36361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 15:16:19.033517 16352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:16:19.033782 16352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:16:19.033910 16352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:16:19.040009 16352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:16:19.040088 16352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:16:19.040202 16352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:16:19.048228 16352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:36361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:16:19.049572 16352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:16:19.051685 16352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:16:19.063488 16358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:19.063446 16360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:19.063539 16357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:16:19.069262 16352 server_base.cc:1034] running on GCE node
I20260926 15:16:19.069818 16352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:16:19.070778 16352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:16:19.072024 16352 hybrid_clock.cc:648] HybridClock initialized: now 1790435779071975 us; error 44 us; skew 500 ppm
I20260926 15:16:19.072320 16352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:16:19.074399 16352 webserver.cc:460] Webserver started at http://127.6.209.193:39381/ using document root <none> and password file <none>
I20260926 15:16:19.075028 16352 fs_manager.cc:359] Metadata directory not provided
I20260926 15:16:19.075162 16352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:16:19.075454 16352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:16:19.078058 16352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "a9f78d742b674ad0be766a72778f5c17"
format_stamp: "Formatted at 2026-09-26 15:16:19 on dist-test-slave-j6q7"
server_key: "1f78c0619bffba5b58f78630deb96985"
server_key_iv: "7594cb40f6c329a44639d744bd7ddbb6"
server_key_version: "encryptionkey@0"
I20260926 15:16:19.078805 16352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "a9f78d742b674ad0be766a72778f5c17"
format_stamp: "Formatted at 2026-09-26 15:16:19 on dist-test-slave-j6q7"
server_key: "1f78c0619bffba5b58f78630deb96985"
server_key_iv: "7594cb40f6c329a44639d744bd7ddbb6"
server_key_version: "encryptionkey@0"
I20260926 15:16:19.083556 16352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 15:16:19.086576 16366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:19.087776 16352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 15:16:19.087970 16352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "a9f78d742b674ad0be766a72778f5c17"
format_stamp: "Formatted at 2026-09-26 15:16:19 on dist-test-slave-j6q7"
server_key: "1f78c0619bffba5b58f78630deb96985"
server_key_iv: "7594cb40f6c329a44639d744bd7ddbb6"
server_key_version: "encryptionkey@0"
I20260926 15:16:19.088162 16352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:16:19.117364 16352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:16:19.118078 16352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:16:19.118355 16352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:16:19.119155 16352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:16:19.120473 16352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:16:19.120613 16352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:19.120759 16352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:16:19.120882 16352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:19.151052 16352 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:36499
I20260926 15:16:19.151139 16478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:36499 every 8 connection(s)
I20260926 15:16:19.152244 16352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:16:19.159140 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 16352
I20260926 15:16:19.159339 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:16:19.159870 6983 external_mini_cluster.cc:1428] Setting key 3552ea4bb1d5907172ddac1af49343af
I20260926 15:16:19.162649 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44969
--builtin_ntp_servers=127.6.209.212:36361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 15:16:19.163527 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:19.154508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:33945 (local address 127.6.209.254:44969)
0926 15:16:19.154890 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:19.154894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:19.155803 (+ 909us) server_negotiation.cc:407] Connection header received
0926 15:16:19.157310 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:19.157315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:19.157389 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:19.157491 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:19.158346 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:19.159115 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:19.160835 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:19.161449 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:19.161583 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:19.162849 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:19.162855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:19.162863 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:16:19.162878 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:19.162917 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:16:19.162928 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:19.162989 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:19.163235 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 15:16:19.163313 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":303,"thread_start_us":212,"threads_started":1}
I20260926 15:16:19.164798 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:19.154948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44969 (local address 127.6.209.193:33945)
0926 15:16:19.155674 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:19.155704 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:19.157095 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:19.157691 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:19.157697 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:19.157768 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:19.158209 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:19.158221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:19.159353 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:19.159357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:19.160657 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:19.160670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:19.161656 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:19.161663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:19.161916 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:19.162534 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:16:19.162537 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:19.162580 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:16:19.162665 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:16:19.163028 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:16:19.163039 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:16:19.163048 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 15:16:19.163223 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 15:16:19.163598 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":228,"threads_started":1}
I20260926 15:16:19.166085 16479 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44969
I20260926 15:16:19.166340 16479 heartbeater.cc:464] Registering TS with master...
I20260926 15:16:19.166960 16479 heartbeater.cc:511] Master 127.6.209.254:44969 requested a full tablet report, sending...
I20260926 15:16:19.168706 16293 ts_manager.cc:194] Registered new tserver with Master: a9f78d742b674ad0be766a72778f5c17 (127.6.209.193:36499)
I20260926 15:16:19.170204 16293 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.209.193:33945
W20260926 15:16:19.447796 16483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:16:19.448015 16483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:16:19.448117 16483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:16:19.454403 16483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:16:19.454473 16483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:16:19.454555 16483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:16:19.462224 16483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:36361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:16:19.463254 16483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:16:19.464699 16483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:16:19.476315 16488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:19.476366 16491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:19.476385 16489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:16:19.476786 16483 server_base.cc:1034] running on GCE node
I20260926 15:16:19.477377 16483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:16:19.478119 16483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:16:19.479323 16483 hybrid_clock.cc:648] HybridClock initialized: now 1790435779479297 us; error 38 us; skew 500 ppm
I20260926 15:16:19.479575 16483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:16:19.481549 16483 webserver.cc:460] Webserver started at http://127.6.209.194:39111/ using document root <none> and password file <none>
I20260926 15:16:19.482157 16483 fs_manager.cc:359] Metadata directory not provided
I20260926 15:16:19.482275 16483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:16:19.482570 16483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:16:19.485119 16483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "2c496be4c7104026910ab3ed2b69518b"
format_stamp: "Formatted at 2026-09-26 15:16:19 on dist-test-slave-j6q7"
server_key: "a5fba0d5b90ca099f9b432b55c87fc28"
server_key_iv: "f33ae30b77b470d8c9a8e1dd797325f4"
server_key_version: "encryptionkey@0"
I20260926 15:16:19.485832 16483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "2c496be4c7104026910ab3ed2b69518b"
format_stamp: "Formatted at 2026-09-26 15:16:19 on dist-test-slave-j6q7"
server_key: "a5fba0d5b90ca099f9b432b55c87fc28"
server_key_iv: "f33ae30b77b470d8c9a8e1dd797325f4"
server_key_version: "encryptionkey@0"
I20260926 15:16:19.490136 16483 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:16:19.493234 16497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:19.494315 16483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:16:19.494467 16483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "2c496be4c7104026910ab3ed2b69518b"
format_stamp: "Formatted at 2026-09-26 15:16:19 on dist-test-slave-j6q7"
server_key: "a5fba0d5b90ca099f9b432b55c87fc28"
server_key_iv: "f33ae30b77b470d8c9a8e1dd797325f4"
server_key_version: "encryptionkey@0"
I20260926 15:16:19.494661 16483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:16:19.506821 16483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:16:19.507496 16483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:16:19.507764 16483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:16:19.508549 16483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:16:19.509814 16483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:16:19.509934 16483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:19.510054 16483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:16:19.510138 16483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:19.538961 16483 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:41739
I20260926 15:16:19.539037 16609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:41739 every 8 connection(s)
I20260926 15:16:19.540030 16483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:16:19.543526 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 16483
I20260926 15:16:19.543689 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:16:19.544076 6983 external_mini_cluster.cc:1428] Setting key 8fd18aff93268ab3d39e189f76add602
I20260926 15:16:19.546618 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44969
--builtin_ntp_servers=127.6.209.212:36361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 15:16:19.551963 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:19.542247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:43961 (local address 127.6.209.254:44969)
0926 15:16:19.542388 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:19.542392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:19.543547 (+ 1155us) server_negotiation.cc:407] Connection header received
0926 15:16:19.545257 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:19.545261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:19.545331 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:19.545422 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:19.546687 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:19.547447 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:19.549483 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:19.549822 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:19.549902 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:19.551321 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:19.551326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:19.551329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:16:19.551344 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:19.551372 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:16:19.551383 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:19.551445 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:19.551709 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 15:16:19.551770 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 15:16:19.553170 16612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:19.542626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44969 (local address 127.6.209.194:43961)
0926 15:16:19.543402 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:19.543431 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:19.545036 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:19.545641 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:19.545650 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:19.545741 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:19.546515 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:19.546539 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:19.547591 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:19.547599 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:19.549370 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:19.549385 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:19.550012 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:19.550020 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:19.550310 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:19.551006 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:16:19.551012 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:19.551059 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:16:19.551145 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:16:19.551469 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:16:19.551481 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:16:19.551491 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 15:16:19.551686 (+ 195us) client_negotiation.cc:240] Negotiation successful
0926 15:16:19.552016 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":232,"threads_started":1}
I20260926 15:16:19.554543 16610 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44969
I20260926 15:16:19.554759 16610 heartbeater.cc:464] Registering TS with master...
I20260926 15:16:19.555382 16610 heartbeater.cc:511] Master 127.6.209.254:44969 requested a full tablet report, sending...
I20260926 15:16:19.556565 16293 ts_manager.cc:194] Registered new tserver with Master: 2c496be4c7104026910ab3ed2b69518b (127.6.209.194:41739)
I20260926 15:16:19.557251 16293 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.209.194:43961
W20260926 15:16:19.833444 16613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:16:19.833722 16613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:16:19.833890 16613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:16:19.840519 16613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:16:19.840652 16613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:16:19.840855 16613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:16:19.848737 16613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:36361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:16:19.849797 16613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:16:19.851362 16613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:16:19.862679 16618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:19.863132 16619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:19.864355 16621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:16:19.864511 16613 server_base.cc:1034] running on GCE node
I20260926 15:16:19.865090 16613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:16:19.865873 16613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:16:19.867177 16613 hybrid_clock.cc:648] HybridClock initialized: now 1790435779867134 us; error 38 us; skew 500 ppm
I20260926 15:16:19.867449 16613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:16:19.869349 16613 webserver.cc:460] Webserver started at http://127.6.209.195:43941/ using document root <none> and password file <none>
I20260926 15:16:19.870003 16613 fs_manager.cc:359] Metadata directory not provided
I20260926 15:16:19.870143 16613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:16:19.870445 16613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:16:19.873239 16613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "d57af233fbfd4e4183b4bd58eff08e3a"
format_stamp: "Formatted at 2026-09-26 15:16:19 on dist-test-slave-j6q7"
server_key: "511eae77ce70a84033d7f0ff43f90726"
server_key_iv: "df3833a0d95fbc1e5c9c7cf07bd5c16b"
server_key_version: "encryptionkey@0"
I20260926 15:16:19.873957 16613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "d57af233fbfd4e4183b4bd58eff08e3a"
format_stamp: "Formatted at 2026-09-26 15:16:19 on dist-test-slave-j6q7"
server_key: "511eae77ce70a84033d7f0ff43f90726"
server_key_iv: "df3833a0d95fbc1e5c9c7cf07bd5c16b"
server_key_version: "encryptionkey@0"
I20260926 15:16:19.878700 16613 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 15:16:19.881716 16627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:19.882907 16613 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:16:19.883067 16613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "d57af233fbfd4e4183b4bd58eff08e3a"
format_stamp: "Formatted at 2026-09-26 15:16:19 on dist-test-slave-j6q7"
server_key: "511eae77ce70a84033d7f0ff43f90726"
server_key_iv: "df3833a0d95fbc1e5c9c7cf07bd5c16b"
server_key_version: "encryptionkey@0"
I20260926 15:16:19.883255 16613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:16:19.896255 16613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:16:19.897037 16613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:16:19.897292 16613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:16:19.898108 16613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:16:19.899389 16613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:16:19.899508 16613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:19.899626 16613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:16:19.899706 16613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:19.929391 16613 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:41631
I20260926 15:16:19.929383 16739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:41631 every 8 connection(s)
I20260926 15:16:19.930824 16613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:16:19.937775 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 16613
I20260926 15:16:19.937930 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:16:19.938422 6983 external_mini_cluster.cc:1428] Setting key 7b34845de45a826a19fddad569d32d0c
I20260926 15:16:19.942636 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:19.933271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:47965 (local address 127.6.209.254:44969)
0926 15:16:19.933421 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:19.933434 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:19.934730 (+ 1296us) server_negotiation.cc:407] Connection header received
0926 15:16:19.936197 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:19.936200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:19.936255 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:19.936337 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:19.937306 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:19.938103 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:19.939635 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:19.940085 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:19.940171 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:19.942042 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:19.942046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:19.942049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:16:19.942087 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:19.942128 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:16:19.942140 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:19.942202 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:19.942366 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 15:16:19.942427 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 15:16:19.943414 16742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:19.933698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44969 (local address 127.6.209.195:47965)
0926 15:16:19.934441 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:19.934475 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:19.935873 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:19.936533 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:19.936540 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:19.936612 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:19.937164 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:19.937178 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:19.938233 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:19.938239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:19.939499 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:19.939516 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:19.940915 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:19.940921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:19.941210 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:19.941810 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:16:19.941812 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:19.941843 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:16:19.941903 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:16:19.942214 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:16:19.942220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:16:19.942225 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:16:19.942341 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 15:16:19.942570 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":233,"threads_started":1}
I20260926 15:16:19.944413 16740 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44969
I20260926 15:16:19.944618 16740 heartbeater.cc:464] Registering TS with master...
I20260926 15:16:19.945245 16740 heartbeater.cc:511] Master 127.6.209.254:44969 requested a full tablet report, sending...
I20260926 15:16:19.946359 16293 ts_manager.cc:194] Registered new tserver with Master: d57af233fbfd4e4183b4bd58eff08e3a (127.6.209.195:41631)
I20260926 15:16:19.947041 16293 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.209.195:47965
I20260926 15:16:19.952781 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 15:16:19.957930 16481 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:19.956584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42028 (local address 127.6.209.254:44969)
0926 15:16:19.956727 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:19.956730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:19.957271 (+ 541us) server_negotiation.cc:407] Connection header received
0926 15:16:19.957382 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:19.957385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:19.957441 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:19.957510 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:19.957777 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42028: BlockingRecv error: recv got EOF from 127.0.0.1:42028 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:16:19.958462 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 16352
I20260926 15:16:19.964926 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:19.959182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60054 (local address 127.6.209.193:36499)
0926 15:16:19.959586 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:19.959597 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:19.959619 (+ 22us) server_negotiation.cc:407] Connection header received
0926 15:16:19.959664 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:19.959668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:19.959817 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:19.959916 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:19.960243 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:19.961263 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:19.962340 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:19.962791 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:19.962876 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:19.963754 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:19.963760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:19.963770 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:16:19.964231 (+ 461us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:19.964269 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:16:19.964285 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:19.964375 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:19.964454 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 15:16:19.964574 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":201,"threads_started":1}
I20260926 15:16:19.965960 16474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:16:20.044734 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 16352
I20260926 15:16:20.060570 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 16483
I20260926 15:16:20.074378 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:20.061397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57840 (local address 127.6.209.194:41739)
0926 15:16:20.061958 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:20.061965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:20.061989 (+ 24us) server_negotiation.cc:407] Connection header received
0926 15:16:20.062048 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:20.062055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:20.062252 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:20.062379 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:20.062699 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:20.063675 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:20.064715 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:20.065254 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:20.065342 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:20.073113 (+ 7771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:20.073121 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:20.073134 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:16:20.073697 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:20.073740 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:16:20.073761 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:20.073848 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:20.073962 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 15:16:20.074071 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":292,"threads_started":1}
I20260926 15:16:20.075529 16605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:16:20.152215 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 16483
I20260926 15:16:20.167137 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 16613
I20260926 15:16:20.174834 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:20.168071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.6.209.195:41631)
0926 15:16:20.168631 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:20.168637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:20.168668 (+ 31us) server_negotiation.cc:407] Connection header received
0926 15:16:20.168726 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:20.168732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:20.168996 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:20.169132 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:20.169498 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:20.170564 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:20.171912 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:20.172466 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:20.172566 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:20.173585 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:20.173593 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:20.173605 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:16:20.174158 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:20.174204 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:16:20.174223 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:20.174308 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:20.174444 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 15:16:20.174552 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":296,"threads_started":1}
I20260926 15:16:20.175973 16735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:16:20.250384 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 16613
I20260926 15:16:20.265281 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 16262
I20260926 15:16:20.266333 16323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:16:20.356681 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 16262
2026-09-26T15:16:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2151 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T15:16:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:16:20Z Disabled control of system clock
I20260926 15:16:20.418581 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44213
--webserver_interface=127.6.209.254
--webserver_port=0
--builtin_ntp_servers=127.6.209.212:42395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:44213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 15:16:20.703147 16759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:16:20.703361 16759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:16:20.703430 16759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:16:20.709683 16759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:16:20.709761 16759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:16:20.709793 16759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:16:20.709842 16759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:16:20.709882 16759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:16:20.717669 16759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:42395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:44213
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:16:20.718652 16759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:16:20.720068 16759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:16:20.730557 16767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:20.730665 16765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:20.730695 16764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:16:20.731374 16759 server_base.cc:1034] running on GCE node
I20260926 15:16:20.732100 16759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:16:20.733371 16759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:16:20.734555 16759 hybrid_clock.cc:648] HybridClock initialized: now 1790435780734520 us; error 28 us; skew 500 ppm
I20260926 15:16:20.734874 16759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:16:20.736896 16759 webserver.cc:460] Webserver started at http://127.6.209.254:37113/ using document root <none> and password file <none>
I20260926 15:16:20.737546 16759 fs_manager.cc:359] Metadata directory not provided
I20260926 15:16:20.737660 16759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:16:20.737967 16759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:16:20.740381 16759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "2f1098704f4849908827ced21b31b433"
format_stamp: "Formatted at 2026-09-26 15:16:20 on dist-test-slave-j6q7"
server_key: "1b2c974d0a81752953c329cfb0bede7a"
server_key_iv: "60fa0faaee3e35c2d65216d094aac6ee"
server_key_version: "encryptionkey@0"
I20260926 15:16:20.741230 16759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "2f1098704f4849908827ced21b31b433"
format_stamp: "Formatted at 2026-09-26 15:16:20 on dist-test-slave-j6q7"
server_key: "1b2c974d0a81752953c329cfb0bede7a"
server_key_iv: "60fa0faaee3e35c2d65216d094aac6ee"
server_key_version: "encryptionkey@0"
I20260926 15:16:20.745775 16759 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:16:20.748696 16773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:20.749861 16759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:16:20.750037 16759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "2f1098704f4849908827ced21b31b433"
format_stamp: "Formatted at 2026-09-26 15:16:20 on dist-test-slave-j6q7"
server_key: "1b2c974d0a81752953c329cfb0bede7a"
server_key_iv: "60fa0faaee3e35c2d65216d094aac6ee"
server_key_version: "encryptionkey@0"
I20260926 15:16:20.750236 16759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:16:20.755287 16759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:16:20.755945 16759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:16:20.756212 16759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:16:20.774432 16759 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:44213
I20260926 15:16:20.774458 16824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:44213 every 8 connection(s)
I20260926 15:16:20.775643 16759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:16:20.778923 16825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:16:20.783332 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 16759
I20260926 15:16:20.783528 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:16:20.784238 6983 external_mini_cluster.cc:1428] Setting key 3106bd6720ab5f0379e903e59a94f450
I20260926 15:16:20.788911 16825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f1098704f4849908827ced21b31b433: Bootstrap starting.
I20260926 15:16:20.791750 16825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f1098704f4849908827ced21b31b433: Neither blocks nor log segments found. Creating new log.
I20260926 15:16:20.791901 16828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:20.786042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33862 (local address 127.6.209.254:44213)
0926 15:16:20.786898 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:20.786907 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:20.786943 (+ 36us) server_negotiation.cc:407] Connection header received
0926 15:16:20.788485 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:20.788506 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:20.788934 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:20.789303 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:20.790279 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:20.790285 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:20.790295 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:16:20.790307 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:20.790383 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:16:20.790399 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:20.790663 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:20.790795 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 15:16:20.791018 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":288,"threads_started":1}
I20260926 15:16:20.792722 16825 log.cc:824] T 00000000000000000000000000000000 P 2f1098704f4849908827ced21b31b433: Log is configured to *not* fsync() on all Append() calls
I20260926 15:16:20.795321 16825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f1098704f4849908827ced21b31b433: No bootstrap required, opened a new log
I20260926 15:16:20.801970 16825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f1098704f4849908827ced21b31b433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1098704f4849908827ced21b31b433" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44213 } }
I20260926 15:16:20.802270 16825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f1098704f4849908827ced21b31b433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:16:20.802362 16825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f1098704f4849908827ced21b31b433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f1098704f4849908827ced21b31b433, State: Initialized, Role: FOLLOWER
I20260926 15:16:20.802883 16825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f1098704f4849908827ced21b31b433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1098704f4849908827ced21b31b433" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44213 } }
I20260926 15:16:20.803128 16825 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f1098704f4849908827ced21b31b433 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:16:20.803263 16825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f1098704f4849908827ced21b31b433 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:16:20.803442 16825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f1098704f4849908827ced21b31b433 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:16:20.805267 16825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f1098704f4849908827ced21b31b433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1098704f4849908827ced21b31b433" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44213 } }
I20260926 15:16:20.805692 16825 leader_election.cc:302] T 00000000000000000000000000000000 P 2f1098704f4849908827ced21b31b433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f1098704f4849908827ced21b31b433; no voters:
I20260926 15:16:20.806186 16825 leader_election.cc:288] T 00000000000000000000000000000000 P 2f1098704f4849908827ced21b31b433 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:16:20.806263 16830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f1098704f4849908827ced21b31b433 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:16:20.806952 16830 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f1098704f4849908827ced21b31b433 [term 1 LEADER]: Becoming Leader. State: Replica: 2f1098704f4849908827ced21b31b433, State: Running, Role: LEADER
I20260926 15:16:20.807456 16830 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f1098704f4849908827ced21b31b433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1098704f4849908827ced21b31b433" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44213 } }
I20260926 15:16:20.808020 16825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f1098704f4849908827ced21b31b433 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:16:20.810163 16832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f1098704f4849908827ced21b31b433 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f1098704f4849908827ced21b31b433. Latest consensus state: current_term: 1 leader_uuid: "2f1098704f4849908827ced21b31b433" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1098704f4849908827ced21b31b433" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44213 } } }
I20260926 15:16:20.810232 16831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f1098704f4849908827ced21b31b433 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f1098704f4849908827ced21b31b433" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1098704f4849908827ced21b31b433" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44213 } } }
I20260926 15:16:20.810544 16831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f1098704f4849908827ced21b31b433 [sys.catalog]: This master's current role is: LEADER
I20260926 15:16:20.810575 16832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f1098704f4849908827ced21b31b433 [sys.catalog]: This master's current role is: LEADER
I20260926 15:16:20.811944 16838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:16:20.816650 16838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:16:20.824404 16838 catalog_manager.cc:1347] Generated new cluster ID: 10151c2910e54dd4a8bd92a59ef4a9f6
I20260926 15:16:20.824497 16838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:16:20.839789 16838 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:16:20.840144 16838 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:16:20.845902 16838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f1098704f4849908827ced21b31b433: Generated new TSK 0
I20260926 15:16:20.846568 16838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:16:20.852476 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44213
--builtin_ntp_servers=127.6.209.212:42395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 15:16:21.148029 16849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:16:21.148272 16849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:16:21.148351 16849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:16:21.154688 16849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:16:21.154770 16849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:16:21.154911 16849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:16:21.162495 16849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:42395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:16:21.163396 16849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:16:21.164881 16849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:16:21.175729 16857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:21.175789 16854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:21.175814 16855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:16:21.176482 16849 server_base.cc:1034] running on GCE node
I20260926 15:16:21.177089 16849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:16:21.177852 16849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:16:21.179024 16849 hybrid_clock.cc:648] HybridClock initialized: now 1790435781178993 us; error 30 us; skew 500 ppm
I20260926 15:16:21.179283 16849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:16:21.181422 16849 webserver.cc:460] Webserver started at http://127.6.209.193:41151/ using document root <none> and password file <none>
I20260926 15:16:21.182132 16849 fs_manager.cc:359] Metadata directory not provided
I20260926 15:16:21.182236 16849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:16:21.182504 16849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:16:21.184753 16849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "dcc54b32358a416eb54efd7e81973689"
format_stamp: "Formatted at 2026-09-26 15:16:21 on dist-test-slave-j6q7"
server_key: "9b403d5258f5e7fe98bf89c3a648968d"
server_key_iv: "7ac8ad0a34b81ff6d71271900b32b0a9"
server_key_version: "encryptionkey@0"
I20260926 15:16:21.185436 16849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "dcc54b32358a416eb54efd7e81973689"
format_stamp: "Formatted at 2026-09-26 15:16:21 on dist-test-slave-j6q7"
server_key: "9b403d5258f5e7fe98bf89c3a648968d"
server_key_iv: "7ac8ad0a34b81ff6d71271900b32b0a9"
server_key_version: "encryptionkey@0"
I20260926 15:16:21.190423 16849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 15:16:21.193418 16863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:21.194456 16849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:16:21.194618 16849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "dcc54b32358a416eb54efd7e81973689"
format_stamp: "Formatted at 2026-09-26 15:16:21 on dist-test-slave-j6q7"
server_key: "9b403d5258f5e7fe98bf89c3a648968d"
server_key_iv: "7ac8ad0a34b81ff6d71271900b32b0a9"
server_key_version: "encryptionkey@0"
I20260926 15:16:21.194806 16849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:16:21.199473 16849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:16:21.200106 16849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:16:21.200337 16849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:16:21.201210 16849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:16:21.202387 16849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:16:21.202525 16849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:21.202678 16849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:16:21.202772 16849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:21.231374 16849 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:41169
I20260926 15:16:21.231447 16975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:41169 every 8 connection(s)
I20260926 15:16:21.232566 16849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:16:21.235360 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 16849
I20260926 15:16:21.235535 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:16:21.236135 6983 external_mini_cluster.cc:1428] Setting key b16a177872dfcdd4b295a3e98c62bca7
I20260926 15:16:21.238497 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44213
--builtin_ntp_servers=127.6.209.212:42395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 15:16:21.240017 16828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:21.234831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:36939 (local address 127.6.209.254:44213)
0926 15:16:21.235033 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:21.235039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:21.236385 (+ 1346us) server_negotiation.cc:407] Connection header received
0926 15:16:21.237917 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:21.237924 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:21.237984 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:21.238105 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:21.239390 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:21.239395 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:21.239399 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:16:21.239415 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:21.239451 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:16:21.239461 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:21.239581 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:21.239737 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 15:16:21.239794 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 15:16:21.240866 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:21.235296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44213 (local address 127.6.209.193:36939)
0926 15:16:21.236217 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:21.236260 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:21.237673 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:21.238308 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:21.238316 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:21.238395 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:21.239140 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:16:21.239142 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:21.239173 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:16:21.239243 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:16:21.239552 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:16:21.239561 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:16:21.239569 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 15:16:21.239804 (+ 235us) client_negotiation.cc:240] Negotiation successful
0926 15:16:21.240000 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":323,"threads_started":1}
I20260926 15:16:21.242195 16976 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44213
I20260926 15:16:21.242429 16976 heartbeater.cc:464] Registering TS with master...
I20260926 15:16:21.242910 16976 heartbeater.cc:511] Master 127.6.209.254:44213 requested a full tablet report, sending...
I20260926 15:16:21.244755 16790 ts_manager.cc:194] Registered new tserver with Master: dcc54b32358a416eb54efd7e81973689 (127.6.209.193:41169)
W20260926 15:16:21.524403 16979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:16:21.524657 16979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:16:21.524731 16979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:16:21.530652 16979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:16:21.530748 16979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:16:21.530844 16979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:16:21.538465 16979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:42395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:16:21.539400 16979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:16:21.540949 16979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:16:21.552556 16985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:21.552687 16984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:21.552649 16987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:16:21.553436 16979 server_base.cc:1034] running on GCE node
I20260926 15:16:21.554041 16979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:16:21.554926 16979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:16:21.556115 16979 hybrid_clock.cc:648] HybridClock initialized: now 1790435781556072 us; error 34 us; skew 500 ppm
I20260926 15:16:21.556430 16979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:16:21.559165 16979 webserver.cc:460] Webserver started at http://127.6.209.194:33985/ using document root <none> and password file <none>
I20260926 15:16:21.559768 16979 fs_manager.cc:359] Metadata directory not provided
I20260926 15:16:21.559901 16979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:16:21.560221 16979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:16:21.564101 16979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "074644bbb28d4f4fac5c4d79e979979e"
format_stamp: "Formatted at 2026-09-26 15:16:21 on dist-test-slave-j6q7"
server_key: "598ec54cf2229620ad8a9b320e7d0ca0"
server_key_iv: "be8708072bbd06061213a0e07cd5d09a"
server_key_version: "encryptionkey@0"
I20260926 15:16:21.564905 16979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "074644bbb28d4f4fac5c4d79e979979e"
format_stamp: "Formatted at 2026-09-26 15:16:21 on dist-test-slave-j6q7"
server_key: "598ec54cf2229620ad8a9b320e7d0ca0"
server_key_iv: "be8708072bbd06061213a0e07cd5d09a"
server_key_version: "encryptionkey@0"
I20260926 15:16:21.569069 16979 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260926 15:16:21.571956 16993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:21.573082 16979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:16:21.573223 16979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "074644bbb28d4f4fac5c4d79e979979e"
format_stamp: "Formatted at 2026-09-26 15:16:21 on dist-test-slave-j6q7"
server_key: "598ec54cf2229620ad8a9b320e7d0ca0"
server_key_iv: "be8708072bbd06061213a0e07cd5d09a"
server_key_version: "encryptionkey@0"
I20260926 15:16:21.573397 16979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:16:21.578105 16979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:16:21.578760 16979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:16:21.578994 16979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:16:21.579885 16979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:16:21.581348 16979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:16:21.581478 16979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:21.581622 16979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:16:21.581722 16979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:21.613019 16979 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:46073
I20260926 15:16:21.613095 17105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:46073 every 8 connection(s)
I20260926 15:16:21.614162 16979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:16:21.617815 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 16979
I20260926 15:16:21.617957 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:16:21.618479 6983 external_mini_cluster.cc:1428] Setting key 73a4ef66d808bc0a87a0b1182457268a
I20260926 15:16:21.621305 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44213
--builtin_ntp_servers=127.6.209.212:42395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 15:16:21.621963 16828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:21.616395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:33459 (local address 127.6.209.254:44213)
0926 15:16:21.616544 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:21.616548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:21.617868 (+ 1320us) server_negotiation.cc:407] Connection header received
0926 15:16:21.619732 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:21.619736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:21.619776 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:21.619839 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:21.621125 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:21.621129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:21.621133 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:16:21.621147 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:21.621175 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:16:21.621186 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:21.621281 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:21.621555 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 15:16:21.621618 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 15:16:21.623445 17108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:21.616869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44213 (local address 127.6.209.194:33459)
0926 15:16:21.617704 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:21.617750 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:21.619475 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:21.620063 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:21.620070 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:21.620144 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:21.620783 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:16:21.620787 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:21.620831 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:16:21.620963 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:16:21.621285 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:16:21.621298 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:16:21.621307 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 15:16:21.622255 (+ 948us) client_negotiation.cc:240] Negotiation successful
0926 15:16:21.622459 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":228,"threads_started":1}
I20260926 15:16:21.624516 17106 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44213
I20260926 15:16:21.624776 17106 heartbeater.cc:464] Registering TS with master...
I20260926 15:16:21.625330 17106 heartbeater.cc:511] Master 127.6.209.254:44213 requested a full tablet report, sending...
I20260926 15:16:21.626603 16790 ts_manager.cc:194] Registered new tserver with Master: 074644bbb28d4f4fac5c4d79e979979e (127.6.209.194:46073)
W20260926 15:16:21.902985 17109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:16:21.903231 17109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:16:21.903391 17109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:16:21.909936 17109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:16:21.910043 17109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:16:21.910176 17109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:16:21.918275 17109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:42395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:16:21.919224 17109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:16:21.920786 17109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:16:21.932405 17115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:21.932590 17117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:21.932451 17114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:16:21.933171 17109 server_base.cc:1034] running on GCE node
I20260926 15:16:21.933646 17109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:16:21.934401 17109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:16:21.935599 17109 hybrid_clock.cc:648] HybridClock initialized: now 1790435781935570 us; error 39 us; skew 500 ppm
I20260926 15:16:21.935853 17109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:16:21.937688 17109 webserver.cc:460] Webserver started at http://127.6.209.195:39363/ using document root <none> and password file <none>
I20260926 15:16:21.938313 17109 fs_manager.cc:359] Metadata directory not provided
I20260926 15:16:21.938426 17109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:16:21.938706 17109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:16:21.941133 17109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "83f5dd76bce4482881db2635712bd9bb"
format_stamp: "Formatted at 2026-09-26 15:16:21 on dist-test-slave-j6q7"
server_key: "992bdbe17dee7f326634e0e7d3aa831c"
server_key_iv: "c2e6d73c8468ea303ad3a277bd02a2e0"
server_key_version: "encryptionkey@0"
I20260926 15:16:21.941866 17109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "83f5dd76bce4482881db2635712bd9bb"
format_stamp: "Formatted at 2026-09-26 15:16:21 on dist-test-slave-j6q7"
server_key: "992bdbe17dee7f326634e0e7d3aa831c"
server_key_iv: "c2e6d73c8468ea303ad3a277bd02a2e0"
server_key_version: "encryptionkey@0"
I20260926 15:16:21.945855 17109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 15:16:21.948730 17123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:21.949735 17109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:16:21.949882 17109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "83f5dd76bce4482881db2635712bd9bb"
format_stamp: "Formatted at 2026-09-26 15:16:21 on dist-test-slave-j6q7"
server_key: "992bdbe17dee7f326634e0e7d3aa831c"
server_key_iv: "c2e6d73c8468ea303ad3a277bd02a2e0"
server_key_version: "encryptionkey@0"
I20260926 15:16:21.950062 17109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:16:21.954892 17109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:16:21.955564 17109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:16:21.955796 17109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:16:21.956538 17109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:16:21.957702 17109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:16:21.957810 17109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:21.957932 17109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:16:21.958012 17109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:21.986466 17109 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:43531
I20260926 15:16:21.986541 17235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:43531 every 8 connection(s)
I20260926 15:16:21.987527 17109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:16:21.991146 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 17109
I20260926 15:16:21.991307 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:16:21.991889 6983 external_mini_cluster.cc:1428] Setting key b301f1cb57c455184c1ecacdf980a936
I20260926 15:16:21.995373 16828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:21.989714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:37345 (local address 127.6.209.254:44213)
0926 15:16:21.989905 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:21.989910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:21.991289 (+ 1379us) server_negotiation.cc:407] Connection header received
0926 15:16:21.993405 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:21.993409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:21.993500 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:21.993606 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:21.994876 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:21.994880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:21.994883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:16:21.994895 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:21.994921 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:16:21.994931 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:21.995016 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:21.995160 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 15:16:21.995199 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 15:16:21.996110 17238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:21.990180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44213 (local address 127.6.209.195:37345)
0926 15:16:21.991112 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:21.991165 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:21.993149 (+ 1984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:21.993851 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:21.993861 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:21.993948 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:21.994578 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:16:21.994580 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:21.994632 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:16:21.994730 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:16:21.995013 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:16:21.995019 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:16:21.995024 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:16:21.995198 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 15:16:21.995349 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":245,"threads_started":1}
I20260926 15:16:21.997366 17236 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44213
I20260926 15:16:21.997617 17236 heartbeater.cc:464] Registering TS with master...
I20260926 15:16:21.998086 17236 heartbeater.cc:511] Master 127.6.209.254:44213 requested a full tablet report, sending...
I20260926 15:16:21.999250 16790 ts_manager.cc:194] Registered new tserver with Master: 83f5dd76bce4482881db2635712bd9bb (127.6.209.195:43531)
I20260926 15:16:22.006388 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 15:16:22.011885 16828 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:16:22.010358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45258 (local address 127.6.209.254:44213)
0926 15:16:22.010478 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:22.010482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:22.010980 (+ 498us) server_negotiation.cc:407] Connection header received
0926 15:16:22.011155 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:22.011168 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:22.011208 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:22.011300 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:22.011644 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45258: BlockingRecv error: recv got EOF from 127.0.0.1:45258 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 15:16:22.012681 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 16849
I20260926 15:16:22.015460 17245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:22.013562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58938 (local address 127.6.209.193:41169)
0926 15:16:22.013957 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:22.013963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:22.013990 (+ 27us) server_negotiation.cc:407] Connection header received
0926 15:16:22.014035 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:22.014039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:22.014174 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:22.014272 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:22.014968 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:22.014972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:22.014985 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:16:22.014999 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:22.015030 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:16:22.015041 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:22.015126 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:22.015193 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 15:16:22.015250 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":197,"threads_started":1}
I20260926 15:16:22.016505 16971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:16:22.091087 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 16849
I20260926 15:16:22.106070 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 16979
I20260926 15:16:22.109738 17247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:22.107009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38450 (local address 127.6.209.194:46073)
0926 15:16:22.107543 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:22.107549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:22.107574 (+ 25us) server_negotiation.cc:407] Connection header received
0926 15:16:22.107631 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:22.107637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:22.107841 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:22.107978 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:22.109032 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:22.109038 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:22.109049 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:16:22.109064 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:22.109100 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:16:22.109113 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:22.109239 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:22.109311 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 15:16:22.109423 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":265,"threads_started":1}
I20260926 15:16:22.110879 17101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:16:22.184403 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 16979
I20260926 15:16:22.198586 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 17109
I20260926 15:16:22.202127 17249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:22.199466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43932 (local address 127.6.209.195:43531)
0926 15:16:22.200026 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:22.200033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:22.200060 (+ 27us) server_negotiation.cc:407] Connection header received
0926 15:16:22.200115 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:22.200121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:22.200319 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:22.200433 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:22.201465 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:22.201470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:22.201481 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:16:22.201496 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:22.201531 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:16:22.201544 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:22.201665 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:22.201713 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 15:16:22.201818 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":270,"threads_started":1}
I20260926 15:16:22.203187 17231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:16:22.277397 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 17109
I20260926 15:16:22.291859 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 16759
I20260926 15:16:22.293011 16820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:16:22.380645 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 16759
2026-09-26T15:16:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2024 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:16:22 dist-test-slave-j6q7 krb5kdc[17254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:16:22 dist-test-slave-j6q7 krb5kdc[17254](info): set up 2 sockets
Sep 26 15:16:22 dist-test-slave-j6q7 krb5kdc[17254](info): commencing operation
krb5kdc: starting...
W20260926 15:16:23.377192 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:16:23 dist-test-slave-j6q7 krb5kdc[17254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:16:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:16:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:16:23.555569 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:35343
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:39087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:35343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:16:23.837927 17267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:16:23.838239 17267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:16:23.838398 17267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:16:23.844861 17267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:16:23.845005 17267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:16:23.845081 17267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:16:23.845157 17267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:16:23.845216 17267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:16:23.853430 17267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:39087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:35343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:35343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:16:23.854535 17267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:16:23.855700 17267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:16:23.866026 17272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:23.866077 17275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:23.866094 17273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:16:23.866611 17267 server_base.cc:1034] running on GCE node
I20260926 15:16:23.867267 17267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:16:23.868425 17267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:16:23.869601 17267 hybrid_clock.cc:648] HybridClock initialized: now 1790435783869564 us; error 28 us; skew 500 ppm
I20260926 15:16:23.869872 17267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:16:23 dist-test-slave-j6q7 krb5kdc[17254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435783, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:16:23.872568 17267 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:16:23.873908 17267 webserver.cc:460] Webserver started at http://127.6.209.254:36207/ using document root <none> and password file <none>
I20260926 15:16:23.874503 17267 fs_manager.cc:359] Metadata directory not provided
I20260926 15:16:23.874619 17267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:16:23.874897 17267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:16:23.877293 17267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "a27f70ee04874ec2afdd5ba57660758d"
format_stamp: "Formatted at 2026-09-26 15:16:23 on dist-test-slave-j6q7"
server_key: "c1b8a907a5a8f7368f64ee1154db458c"
server_key_iv: "5e0674c5e9e7ea615ba097a46ec5cdf1"
server_key_version: "encryptionkey@0"
I20260926 15:16:23.877944 17267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "a27f70ee04874ec2afdd5ba57660758d"
format_stamp: "Formatted at 2026-09-26 15:16:23 on dist-test-slave-j6q7"
server_key: "c1b8a907a5a8f7368f64ee1154db458c"
server_key_iv: "5e0674c5e9e7ea615ba097a46ec5cdf1"
server_key_version: "encryptionkey@0"
I20260926 15:16:23.882604 17267 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 15:16:23.885497 17282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:23.886523 17267 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:16:23.886672 17267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "a27f70ee04874ec2afdd5ba57660758d"
format_stamp: "Formatted at 2026-09-26 15:16:23 on dist-test-slave-j6q7"
server_key: "c1b8a907a5a8f7368f64ee1154db458c"
server_key_iv: "5e0674c5e9e7ea615ba097a46ec5cdf1"
server_key_version: "encryptionkey@0"
I20260926 15:16:23.886808 17267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:16:23.900445 17267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:16:23.903836 17267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:16:23.904072 17267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:16:23.919683 17267 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:35343
I20260926 15:16:23.919750 17333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:35343 every 8 connection(s)
I20260926 15:16:23.920956 17267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:16:23.923954 17334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:16:23.930552 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 17267
I20260926 15:16:23.930719 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:16:23.931463 6983 external_mini_cluster.cc:1428] Setting key eb92832d8f82dd1ca54ec43b7ef16fa6
I20260926 15:16:23.933099 17334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a27f70ee04874ec2afdd5ba57660758d: Bootstrap starting.
I20260926 15:16:23.935758 17334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a27f70ee04874ec2afdd5ba57660758d: Neither blocks nor log segments found. Creating new log.
I20260926 15:16:23.936929 17334 log.cc:824] T 00000000000000000000000000000000 P a27f70ee04874ec2afdd5ba57660758d: Log is configured to *not* fsync() on all Append() calls
I20260926 15:16:23.939400 17334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a27f70ee04874ec2afdd5ba57660758d: No bootstrap required, opened a new log
Sep 26 15:16:23 dist-test-slave-j6q7 krb5kdc[17254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:16:23.947125 17334 raft_consensus.cc:348] T 00000000000000000000000000000000 P a27f70ee04874ec2afdd5ba57660758d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27f70ee04874ec2afdd5ba57660758d" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35343 } }
I20260926 15:16:23.947425 17334 raft_consensus.cc:374] T 00000000000000000000000000000000 P a27f70ee04874ec2afdd5ba57660758d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:16:23.947538 17334 raft_consensus.cc:729] T 00000000000000000000000000000000 P a27f70ee04874ec2afdd5ba57660758d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a27f70ee04874ec2afdd5ba57660758d, State: Initialized, Role: FOLLOWER
I20260926 15:16:23.948031 17334 consensus_queue.cc:260] T 00000000000000000000000000000000 P a27f70ee04874ec2afdd5ba57660758d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27f70ee04874ec2afdd5ba57660758d" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35343 } }
I20260926 15:16:23.948278 17334 raft_consensus.cc:388] T 00000000000000000000000000000000 P a27f70ee04874ec2afdd5ba57660758d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:16:23.948390 17334 raft_consensus.cc:482] T 00000000000000000000000000000000 P a27f70ee04874ec2afdd5ba57660758d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:16:23.948575 17334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a27f70ee04874ec2afdd5ba57660758d [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:16:23.950920 17334 raft_consensus.cc:504] T 00000000000000000000000000000000 P a27f70ee04874ec2afdd5ba57660758d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27f70ee04874ec2afdd5ba57660758d" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35343 } }
I20260926 15:16:23.951206 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:23.933275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57340 (local address 127.6.209.254:35343)
0926 15:16:23.933960 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:23.933976 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:23.934007 (+ 31us) server_negotiation.cc:407] Connection header received
0926 15:16:23.935418 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:23.935436 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:23.935724 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:23.936044 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:23.937614 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:23.938873 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:23.940367 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:23.941067 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:23.941189 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:23.945068 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:23.945075 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:23.945086 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:16:23.945120 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:23.947779 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:23.948463 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:23.948470 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:23.948474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:23.948651 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:23.949027 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:23.949030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:23.949032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:23.949454 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:23.949667 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:23.950009 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 15:16:23.950297 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":324,"threads_started":1}
I20260926 15:16:23.951457 17334 leader_election.cc:302] T 00000000000000000000000000000000 P a27f70ee04874ec2afdd5ba57660758d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a27f70ee04874ec2afdd5ba57660758d; no voters:
I20260926 15:16:23.952066 17334 leader_election.cc:288] T 00000000000000000000000000000000 P a27f70ee04874ec2afdd5ba57660758d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:16:23.952180 17339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a27f70ee04874ec2afdd5ba57660758d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:16:23.953902 17334 sys_catalog.cc:564] T 00000000000000000000000000000000 P a27f70ee04874ec2afdd5ba57660758d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:16:23.957106 17339 raft_consensus.cc:686] T 00000000000000000000000000000000 P a27f70ee04874ec2afdd5ba57660758d [term 1 LEADER]: Becoming Leader. State: Replica: a27f70ee04874ec2afdd5ba57660758d, State: Running, Role: LEADER
I20260926 15:16:23.957567 17339 consensus_queue.cc:237] T 00000000000000000000000000000000 P a27f70ee04874ec2afdd5ba57660758d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27f70ee04874ec2afdd5ba57660758d" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35343 } }
I20260926 15:16:23.959851 17341 sys_catalog.cc:455] T 00000000000000000000000000000000 P a27f70ee04874ec2afdd5ba57660758d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a27f70ee04874ec2afdd5ba57660758d. Latest consensus state: current_term: 1 leader_uuid: "a27f70ee04874ec2afdd5ba57660758d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27f70ee04874ec2afdd5ba57660758d" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35343 } } }
I20260926 15:16:23.960275 17341 sys_catalog.cc:458] T 00000000000000000000000000000000 P a27f70ee04874ec2afdd5ba57660758d [sys.catalog]: This master's current role is: LEADER
I20260926 15:16:23.960879 17340 sys_catalog.cc:455] T 00000000000000000000000000000000 P a27f70ee04874ec2afdd5ba57660758d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a27f70ee04874ec2afdd5ba57660758d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27f70ee04874ec2afdd5ba57660758d" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35343 } } }
I20260926 15:16:23.961216 17340 sys_catalog.cc:458] T 00000000000000000000000000000000 P a27f70ee04874ec2afdd5ba57660758d [sys.catalog]: This master's current role is: LEADER
I20260926 15:16:23.961825 17350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:16:23.966166 17350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:16:23.973754 17350 catalog_manager.cc:1347] Generated new cluster ID: b409f14f92da47108d6ac7272a691079
I20260926 15:16:23.973837 17350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:16:24.006408 17350 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:16:24.007489 17350 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:16:24.018317 17350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a27f70ee04874ec2afdd5ba57660758d: Generated new TSK 0
I20260926 15:16:24.019150 17350 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
I20260926 15:16:24.083918 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35343
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:39087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:16:24.403888 17360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:16:24.404121 17360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:16:24.404292 17360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:16:24.410485 17360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:16:24.410599 17360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:16:24.410775 17360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:16:24.419030 17360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:39087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:16:24.420022 17360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:16:24.421241 17360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:16:24.432842 17366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:24.432909 17365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:24.432929 17368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:16:24.434044 17360 server_base.cc:1034] running on GCE node
I20260926 15:16:24.434558 17360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:16:24.435339 17360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:16:24.436535 17360 hybrid_clock.cc:648] HybridClock initialized: now 1790435784436501 us; error 24 us; skew 500 ppm
I20260926 15:16:24.436928 17360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:16:24 dist-test-slave-j6q7 krb5kdc[17254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435784, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:16:24.439908 17360 init.cc:377] Logged in from keytab as kudu/127.6.209.193@KRBTEST.COM (short username kudu)
I20260926 15:16:24.441248 17360 webserver.cc:460] Webserver started at http://127.6.209.193:36909/ using document root <none> and password file <none>
I20260926 15:16:24.441883 17360 fs_manager.cc:359] Metadata directory not provided
I20260926 15:16:24.441990 17360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:16:24.442240 17360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:16:24.444655 17360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "9a94564df381458faad3c21d44ba4c39"
format_stamp: "Formatted at 2026-09-26 15:16:24 on dist-test-slave-j6q7"
server_key: "fa81698926d52f08d6a83420877e8fc9"
server_key_iv: "2e54ef04d85624ef9969eb26c2a74c28"
server_key_version: "encryptionkey@0"
I20260926 15:16:24.445369 17360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "9a94564df381458faad3c21d44ba4c39"
format_stamp: "Formatted at 2026-09-26 15:16:24 on dist-test-slave-j6q7"
server_key: "fa81698926d52f08d6a83420877e8fc9"
server_key_iv: "2e54ef04d85624ef9969eb26c2a74c28"
server_key_version: "encryptionkey@0"
I20260926 15:16:24.449776 17360 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:16:24.452790 17375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:24.454008 17360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:16:24.454211 17360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "9a94564df381458faad3c21d44ba4c39"
format_stamp: "Formatted at 2026-09-26 15:16:24 on dist-test-slave-j6q7"
server_key: "fa81698926d52f08d6a83420877e8fc9"
server_key_iv: "2e54ef04d85624ef9969eb26c2a74c28"
server_key_version: "encryptionkey@0"
I20260926 15:16:24.454381 17360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:16:24.466488 17360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:16:24.469880 17360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:16:24.470131 17360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:16:24.470858 17360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:16:24.472002 17360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:16:24.472100 17360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:24.472244 17360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:16:24.472325 17360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:24.500519 17360 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:46489
I20260926 15:16:24.500569 17487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:46489 every 8 connection(s)
I20260926 15:16:24.501721 17360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:16:24.502908 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 17360
I20260926 15:16:24.503129 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:16:24.503835 6983 external_mini_cluster.cc:1428] Setting key d0ab43a30cff0522fc821e0aad54a5e3
Sep 26 15:16:24 dist-test-slave-j6q7 krb5kdc[17254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435784, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:16:24.524130 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:24.504194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:35027 (local address 127.6.209.254:35343)
0926 15:16:24.504619 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:24.504641 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:24.505651 (+ 1010us) server_negotiation.cc:407] Connection header received
0926 15:16:24.508089 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:24.508093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:24.508167 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:24.508248 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:24.510139 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:24.511159 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:24.513621 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:24.514228 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:24.514340 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:24.517956 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:24.517971 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:24.517977 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:16:24.518019 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:24.521244 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:24.521887 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:24.521896 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:24.521902 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:24.522049 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:24.522429 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:24.522436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:24.522442 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:24.522783 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:24.522918 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:24.523494 (+ 576us) server_negotiation.cc:299] Negotiation successful
0926 15:16:24.523717 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":218,"threads_started":1}
I20260926 15:16:24.525043 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:24.504767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:35343 (local address 127.6.209.193:35027)
0926 15:16:24.505507 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:24.505547 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:24.507835 (+ 2288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:24.508515 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:24.508527 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:24.509252 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:24.509959 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:24.509992 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:24.511320 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:24.511328 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:24.513481 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:24.513498 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:24.514449 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:24.514455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:24.514763 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:24.515469 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:24.515491 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:24.517705 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:16:24.521362 (+ 3657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:16:24.521367 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:16:24.521378 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:16:24.521757 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:16:24.522171 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:16:24.522173 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:16:24.522175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:16:24.522320 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:16:24.522888 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:16:24.522893 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:16:24.523136 (+ 243us) client_negotiation.cc:769] Sending connection context
0926 15:16:24.523376 (+ 240us) client_negotiation.cc:240] Negotiation successful
0926 15:16:24.523719 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":262,"threads_started":1}
I20260926 15:16:24.526548 17488 heartbeater.cc:347] Connected to a master server at 127.6.209.254:35343
I20260926 15:16:24.526861 17488 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.209.194@KRBTEST.COM; defaulting to no policy
I20260926 15:16:24.527614 17488 heartbeater.cc:511] Master 127.6.209.254:35343 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.194@KRBTEST.COM" created.
I20260926 15:16:24.529739 17299 ts_manager.cc:194] Registered new tserver with Master: 9a94564df381458faad3c21d44ba4c39 (127.6.209.193:46489)
I20260926 15:16:24.531052 17299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.193@KRBTEST.COM'} at 127.6.209.193:35027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
I20260926 15:16:24.551249 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35343
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:39087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:16:24.832293 17494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:16:24.832553 17494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:16:24.832643 17494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:16:24.838943 17494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:16:24.839049 17494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:16:24.839151 17494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:16:24.847064 17494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:39087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:16:24.847966 17494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:16:24.849088 17494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:16:24.860343 17499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:24.860365 17500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:24.860499 17502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:16:24.861099 17494 server_base.cc:1034] running on GCE node
I20260926 15:16:24.861562 17494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:16:24.862336 17494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:16:24.863528 17494 hybrid_clock.cc:648] HybridClock initialized: now 1790435784863491 us; error 48 us; skew 500 ppm
I20260926 15:16:24.863744 17494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:16:24 dist-test-slave-j6q7 krb5kdc[17254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435784, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:16:24.866279 17494 init.cc:377] Logged in from keytab as kudu/127.6.209.194@KRBTEST.COM (short username kudu)
I20260926 15:16:24.867383 17494 webserver.cc:460] Webserver started at http://127.6.209.194:39609/ using document root <none> and password file <none>
I20260926 15:16:24.867920 17494 fs_manager.cc:359] Metadata directory not provided
I20260926 15:16:24.868065 17494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:16:24.868314 17494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:16:24.870662 17494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "a46d74955b29468281f1856f2db0863c"
format_stamp: "Formatted at 2026-09-26 15:16:24 on dist-test-slave-j6q7"
server_key: "1ad463aa461925200ed421b4f8298957"
server_key_iv: "5ae69ecbb4def92c227b2b61fbe9920b"
server_key_version: "encryptionkey@0"
I20260926 15:16:24.871394 17494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "a46d74955b29468281f1856f2db0863c"
format_stamp: "Formatted at 2026-09-26 15:16:24 on dist-test-slave-j6q7"
server_key: "1ad463aa461925200ed421b4f8298957"
server_key_iv: "5ae69ecbb4def92c227b2b61fbe9920b"
server_key_version: "encryptionkey@0"
I20260926 15:16:24.875476 17494 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 15:16:24.878206 17509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:24.879236 17494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 15:16:24.879380 17494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "a46d74955b29468281f1856f2db0863c"
format_stamp: "Formatted at 2026-09-26 15:16:24 on dist-test-slave-j6q7"
server_key: "1ad463aa461925200ed421b4f8298957"
server_key_iv: "5ae69ecbb4def92c227b2b61fbe9920b"
server_key_version: "encryptionkey@0"
I20260926 15:16:24.879555 17494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:16:24.899787 17494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:16:24.903277 17494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:16:24.903566 17494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:16:24.904381 17494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:16:24.905668 17494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:16:24.905776 17494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:24.905936 17494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:16:24.906015 17494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:24.935456 17494 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:46803
I20260926 15:16:24.935470 17621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:46803 every 8 connection(s)
I20260926 15:16:24.936749 17494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:16:24.937150 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 17494
I20260926 15:16:24.937328 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:16:24.937806 6983 external_mini_cluster.cc:1428] Setting key 30fe49806c330f0a24fe0b9ed203a37d
Sep 26 15:16:24 dist-test-slave-j6q7 krb5kdc[17254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435784, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:16:24.958881 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:24.939058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:34161 (local address 127.6.209.254:35343)
0926 15:16:24.939216 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:24.939221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:24.940279 (+ 1058us) server_negotiation.cc:407] Connection header received
0926 15:16:24.943029 (+ 2750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:24.943034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:24.943106 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:24.943167 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:24.945131 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:24.946116 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:24.948382 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:24.948738 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:24.948941 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:24.952239 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:24.952248 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:24.952256 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:16:24.952298 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:24.955640 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:24.956479 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:24.956486 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:24.956492 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:24.956645 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:24.957176 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:24.957184 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:24.957187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:24.957552 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:24.957731 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:24.958344 (+ 613us) server_negotiation.cc:299] Negotiation successful
0926 15:16:24.958538 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:16:24.959666 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:24.939529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:35343 (local address 127.6.209.194:34161)
0926 15:16:24.940159 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:24.940197 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:24.942722 (+ 2525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:24.943610 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:24.943621 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:24.944265 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:24.944960 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:24.944982 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:24.946319 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:24.946326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:24.948238 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:24.948253 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:24.949011 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:24.949019 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:24.949244 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:24.949827 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:24.949847 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:24.952023 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:16:24.955777 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:16:24.955783 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:16:24.955792 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:16:24.956328 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:16:24.956762 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:16:24.956765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:16:24.956767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:16:24.957053 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:16:24.957697 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:16:24.957705 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:16:24.957982 (+ 277us) client_negotiation.cc:769] Sending connection context
0926 15:16:24.958231 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 15:16:24.958666 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":5357,"threads_started":1}
I20260926 15:16:24.961212 17622 heartbeater.cc:347] Connected to a master server at 127.6.209.254:35343
I20260926 15:16:24.961519 17622 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.209.195@KRBTEST.COM; defaulting to no policy
I20260926 15:16:24.962338 17622 heartbeater.cc:511] Master 127.6.209.254:35343 requested a full tablet report, sending...
I20260926 15:16:24.963704 17299 ts_manager.cc:194] Registered new tserver with Master: a46d74955b29468281f1856f2db0863c (127.6.209.194:46803)
I20260926 15:16:24.964355 17299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.194@KRBTEST.COM'} at 127.6.209.194:34161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
I20260926 15:16:24.985728 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35343
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:39087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:16:25.270366 17627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:16:25.270574 17627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:16:25.270650 17627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:16:25.276750 17627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:16:25.276847 17627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:16:25.276999 17627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:16:25.285949 17627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:39087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:16:25.286867 17627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:16:25.287904 17627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:16:25.299067 17635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:25.299199 17632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:16:25.299163 17633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:16:25.300132 17627 server_base.cc:1034] running on GCE node
I20260926 15:16:25.300724 17627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:16:25.301559 17627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:16:25.302783 17627 hybrid_clock.cc:648] HybridClock initialized: now 1790435785302747 us; error 48 us; skew 500 ppm
I20260926 15:16:25.303072 17627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:16:25 dist-test-slave-j6q7 krb5kdc[17254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435785, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:16:25.313510 17627 init.cc:377] Logged in from keytab as kudu/127.6.209.195@KRBTEST.COM (short username kudu)
I20260926 15:16:25.314703 17627 webserver.cc:460] Webserver started at http://127.6.209.195:37777/ using document root <none> and password file <none>
I20260926 15:16:25.315229 17627 fs_manager.cc:359] Metadata directory not provided
I20260926 15:16:25.315311 17627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:16:25.315515 17627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:16:25.318013 17627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "30cad260cf9446968a17b19a2b3a139a"
format_stamp: "Formatted at 2026-09-26 15:16:25 on dist-test-slave-j6q7"
server_key: "f5cf05345a4392e98a1a0e825d27f5b9"
server_key_iv: "579d7fe7985c33f7756a9495504c98c1"
server_key_version: "encryptionkey@0"
I20260926 15:16:25.318631 17627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "30cad260cf9446968a17b19a2b3a139a"
format_stamp: "Formatted at 2026-09-26 15:16:25 on dist-test-slave-j6q7"
server_key: "f5cf05345a4392e98a1a0e825d27f5b9"
server_key_iv: "579d7fe7985c33f7756a9495504c98c1"
server_key_version: "encryptionkey@0"
I20260926 15:16:25.322983 17627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 15:16:25.325707 17642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:25.326866 17627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:16:25.327024 17627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "30cad260cf9446968a17b19a2b3a139a"
format_stamp: "Formatted at 2026-09-26 15:16:25 on dist-test-slave-j6q7"
server_key: "f5cf05345a4392e98a1a0e825d27f5b9"
server_key_iv: "579d7fe7985c33f7756a9495504c98c1"
server_key_version: "encryptionkey@0"
I20260926 15:16:25.327186 17627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:16:25.353564 17627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:16:25.356745 17627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:16:25.357043 17627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:16:25.357859 17627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:16:25.358914 17627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:16:25.358976 17627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:25.359119 17627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:16:25.359181 17627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:16:25.388046 17627 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:39345
I20260926 15:16:25.388113 17754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:39345 every 8 connection(s)
I20260926 15:16:25.389145 17627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:16:25.390874 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 17627
I20260926 15:16:25.391033 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:16:25.391597 6983 external_mini_cluster.cc:1428] Setting key dfe52f1e7069b8c3a03024a8770ddf93
Sep 26 15:16:25 dist-test-slave-j6q7 krb5kdc[17254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435785, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:16:25.405632 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:25.391444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:48683 (local address 127.6.209.254:35343)
0926 15:16:25.391637 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:25.391643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:25.392827 (+ 1184us) server_negotiation.cc:407] Connection header received
0926 15:16:25.394406 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:25.394409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:25.394486 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:25.394568 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:25.395697 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.396335 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.397819 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.398177 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.398258 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:25.401030 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:25.401034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:25.401037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:16:25.401066 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:25.403190 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:25.403715 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:25.403718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:25.403720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:25.403813 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:25.404109 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:25.404111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:25.404113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:25.404353 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:25.404477 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:25.405162 (+ 685us) server_negotiation.cc:299] Negotiation successful
0926 15:16:25.405347 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 15:16:25.406337 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:25.391980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:35343 (local address 127.6.209.195:48683)
0926 15:16:25.392682 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:25.392729 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:25.394191 (+ 1462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:25.394737 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:25.394744 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:25.395125 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:16:25.395581 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:25.395596 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.396461 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.396473 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:25.397712 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:25.397723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.398327 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.398331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:25.398505 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:25.398975 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:16:25.398994 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:16:25.400792 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:16:25.403295 (+ 2503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:16:25.403300 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:16:25.403308 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:16:25.403623 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:16:25.403886 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:16:25.403889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:16:25.403890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:16:25.404027 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:16:25.404453 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:16:25.404457 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:16:25.404759 (+ 302us) client_negotiation.cc:769] Sending connection context
0926 15:16:25.405053 (+ 294us) client_negotiation.cc:240] Negotiation successful
0926 15:16:25.405436 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":335,"threads_started":1}
I20260926 15:16:25.407364 17755 heartbeater.cc:347] Connected to a master server at 127.6.209.254:35343
I20260926 15:16:25.407617 17755 heartbeater.cc:464] Registering TS with master...
I20260926 15:16:25.408252 17755 heartbeater.cc:511] Master 127.6.209.254:35343 requested a full tablet report, sending...
I20260926 15:16:25.409373 17299 ts_manager.cc:194] Registered new tserver with Master: 30cad260cf9446968a17b19a2b3a139a (127.6.209.195:39345)
I20260926 15:16:25.410004 17299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.195@KRBTEST.COM'} at 127.6.209.195:48683
I20260926 15:16:25.417783 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:16:25.432251 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:25.421757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57346 (local address 127.6.209.254:35343)
0926 15:16:25.421889 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:25.421893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:25.422344 (+ 451us) server_negotiation.cc:407] Connection header received
0926 15:16:25.422505 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:25.422509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:25.422563 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:25.422649 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:25.423764 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.424343 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.425401 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.425768 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.425911 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:25.427810 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:25.427814 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:25.427817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:16:25.427839 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:25.429971 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:25.430477 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:25.430483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:25.430489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:25.430587 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:25.430934 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:25.430940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:25.430943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:25.431221 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:25.431452 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:25.431776 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 15:16:25.431965 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 15:16:25.435428 17299 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:16:25.441159 17299 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:16:25.473057 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:25.451473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36038 (local address 127.6.209.193:46489)
0926 15:16:25.465178 (+ 13705us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:25.465186 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:25.465510 (+ 324us) server_negotiation.cc:407] Connection header received
0926 15:16:25.465740 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:25.465746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:25.465963 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:25.467184 (+ 1221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:16:25.467370 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.468556 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.470623 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.472020 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.472154 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:25.472341 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:25.472608 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 15:16:25.472741 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13563,"thread_start_us":13450,"threads_started":1}
I20260926 15:16:25.473105 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:25.451303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:46489 (local address 127.0.0.1:36038)
0926 15:16:25.465393 (+ 14090us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:25.465405 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:25.465543 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:25.466283 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:25.466291 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:25.466327 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:16:25.466904 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:25.466918 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.468667 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.468672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:25.470486 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:25.470498 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.472244 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.472249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:25.472401 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:25.472442 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 15:16:25.472554 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 15:16:25.472693 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14001,"mutex_wait_us":4,"spinlock_wait_cycles":4864,"thread_start_us":13530,"threads_started":1}
I20260926 15:16:25.473835 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:25.451223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51706 (local address 127.6.209.195:39345)
0926 15:16:25.465510 (+ 14287us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:25.465517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:25.465542 (+ 25us) server_negotiation.cc:407] Connection header received
0926 15:16:25.465624 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:25.465631 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:25.465845 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:25.466057 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:16:25.467518 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.469242 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.471330 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.472588 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.472669 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:25.472866 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:25.473397 (+ 531us) server_negotiation.cc:299] Negotiation successful
0926 15:16:25.473547 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13520,"thread_start_us":13354,"threads_started":1}
I20260926 15:16:25.474399 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:25.450937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:39345 (local address 127.0.0.1:51706)
0926 15:16:25.464068 (+ 13131us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:25.464113 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:25.464275 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:25.466317 (+ 2042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:25.466324 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:25.466354 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:16:25.467293 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:25.467305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.469380 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.469382 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:25.471158 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:25.471171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.473025 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.473031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:25.473217 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:25.473240 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 15:16:25.473983 (+ 743us) client_negotiation.cc:240] Negotiation successful
0926 15:16:25.474096 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12890,"mutex_wait_us":469,"thread_start_us":14236,"threads_started":1}
I20260926 15:16:25.475755 17771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:25.465678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:46803 (local address 127.0.0.1:50520)
0926 15:16:25.466988 (+ 1310us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:25.467000 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:25.467069 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:25.469583 (+ 2514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:25.469587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:25.469613 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:16:25.470100 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:25.470110 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.471402 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.471410 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:25.473586 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:25.473599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.475064 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.475070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:25.475222 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:25.475256 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 15:16:25.475348 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 15:16:25.475453 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1205,"mutex_wait_us":77,"thread_start_us":1062,"threads_started":1}
I20260926 15:16:25.475788 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:25.465739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50520 (local address 127.6.209.194:46803)
0926 15:16:25.469147 (+ 3408us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:25.469155 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:25.469176 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:16:25.469250 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:25.469256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:25.469426 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:25.469579 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:16:25.470255 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.471274 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.473737 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.474870 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.475014 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:25.475176 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:25.475399 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 15:16:25.475547 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3282,"thread_start_us":3032,"threads_started":1}
I20260926 15:16:25.476768 17423 tablet_service.cc:1467] Processing CreateTablet for tablet 230def51d8874b26aaaf211e361adbc2 (DEFAULT_TABLE table=test-table [id=cf48701156b348aab0ee420e21ac1b3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:16:25.477478 17690 tablet_service.cc:1467] Processing CreateTablet for tablet 230def51d8874b26aaaf211e361adbc2 (DEFAULT_TABLE table=test-table [id=cf48701156b348aab0ee420e21ac1b3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:16:25.478085 17423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 230def51d8874b26aaaf211e361adbc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:16:25.478660 17690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 230def51d8874b26aaaf211e361adbc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:16:25.478456 17557 tablet_service.cc:1467] Processing CreateTablet for tablet 230def51d8874b26aaaf211e361adbc2 (DEFAULT_TABLE table=test-table [id=cf48701156b348aab0ee420e21ac1b3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:16:25.479379 17557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 230def51d8874b26aaaf211e361adbc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:16:25.489743 17773 tablet_bootstrap.cc:492] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c: Bootstrap starting.
I20260926 15:16:25.491803 17774 tablet_bootstrap.cc:492] T 230def51d8874b26aaaf211e361adbc2 P 30cad260cf9446968a17b19a2b3a139a: Bootstrap starting.
I20260926 15:16:25.492672 17775 tablet_bootstrap.cc:492] T 230def51d8874b26aaaf211e361adbc2 P 9a94564df381458faad3c21d44ba4c39: Bootstrap starting.
I20260926 15:16:25.493325 17773 tablet_bootstrap.cc:654] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c: Neither blocks nor log segments found. Creating new log.
I20260926 15:16:25.494515 17773 log.cc:824] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c: Log is configured to *not* fsync() on all Append() calls
I20260926 15:16:25.495416 17774 tablet_bootstrap.cc:654] T 230def51d8874b26aaaf211e361adbc2 P 30cad260cf9446968a17b19a2b3a139a: Neither blocks nor log segments found. Creating new log.
I20260926 15:16:25.495532 17775 tablet_bootstrap.cc:654] T 230def51d8874b26aaaf211e361adbc2 P 9a94564df381458faad3c21d44ba4c39: Neither blocks nor log segments found. Creating new log.
I20260926 15:16:25.496606 17775 log.cc:824] T 230def51d8874b26aaaf211e361adbc2 P 9a94564df381458faad3c21d44ba4c39: Log is configured to *not* fsync() on all Append() calls
I20260926 15:16:25.496619 17774 log.cc:824] T 230def51d8874b26aaaf211e361adbc2 P 30cad260cf9446968a17b19a2b3a139a: Log is configured to *not* fsync() on all Append() calls
I20260926 15:16:25.498232 17773 tablet_bootstrap.cc:492] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c: No bootstrap required, opened a new log
I20260926 15:16:25.498524 17773 ts_tablet_manager.cc:1397] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260926 15:16:25.499325 17775 tablet_bootstrap.cc:492] T 230def51d8874b26aaaf211e361adbc2 P 9a94564df381458faad3c21d44ba4c39: No bootstrap required, opened a new log
I20260926 15:16:25.499559 17775 ts_tablet_manager.cc:1397] T 230def51d8874b26aaaf211e361adbc2 P 9a94564df381458faad3c21d44ba4c39: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 15:16:25.503196 17774 tablet_bootstrap.cc:492] T 230def51d8874b26aaaf211e361adbc2 P 30cad260cf9446968a17b19a2b3a139a: No bootstrap required, opened a new log
I20260926 15:16:25.503404 17774 ts_tablet_manager.cc:1397] T 230def51d8874b26aaaf211e361adbc2 P 30cad260cf9446968a17b19a2b3a139a: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260926 15:16:25.507573 17773 raft_consensus.cc:348] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30cad260cf9446968a17b19a2b3a139a" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39345 } } peers { permanent_uuid: "9a94564df381458faad3c21d44ba4c39" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46489 } } peers { permanent_uuid: "a46d74955b29468281f1856f2db0863c" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 46803 } }
I20260926 15:16:25.508051 17773 raft_consensus.cc:374] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:16:25.508167 17773 raft_consensus.cc:729] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a46d74955b29468281f1856f2db0863c, State: Initialized, Role: FOLLOWER
I20260926 15:16:25.508134 17775 raft_consensus.cc:348] T 230def51d8874b26aaaf211e361adbc2 P 9a94564df381458faad3c21d44ba4c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30cad260cf9446968a17b19a2b3a139a" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39345 } } peers { permanent_uuid: "9a94564df381458faad3c21d44ba4c39" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46489 } } peers { permanent_uuid: "a46d74955b29468281f1856f2db0863c" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 46803 } }
I20260926 15:16:25.508636 17775 raft_consensus.cc:374] T 230def51d8874b26aaaf211e361adbc2 P 9a94564df381458faad3c21d44ba4c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:16:25.508780 17775 raft_consensus.cc:729] T 230def51d8874b26aaaf211e361adbc2 P 9a94564df381458faad3c21d44ba4c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a94564df381458faad3c21d44ba4c39, State: Initialized, Role: FOLLOWER
I20260926 15:16:25.508852 17773 consensus_queue.cc:260] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30cad260cf9446968a17b19a2b3a139a" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39345 } } peers { permanent_uuid: "9a94564df381458faad3c21d44ba4c39" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46489 } } peers { permanent_uuid: "a46d74955b29468281f1856f2db0863c" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 46803 } }
I20260926 15:16:25.509446 17775 consensus_queue.cc:260] T 230def51d8874b26aaaf211e361adbc2 P 9a94564df381458faad3c21d44ba4c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30cad260cf9446968a17b19a2b3a139a" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39345 } } peers { permanent_uuid: "9a94564df381458faad3c21d44ba4c39" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46489 } } peers { permanent_uuid: "a46d74955b29468281f1856f2db0863c" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 46803 } }
I20260926 15:16:25.517117 17774 raft_consensus.cc:348] T 230def51d8874b26aaaf211e361adbc2 P 30cad260cf9446968a17b19a2b3a139a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30cad260cf9446968a17b19a2b3a139a" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39345 } } peers { permanent_uuid: "9a94564df381458faad3c21d44ba4c39" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46489 } } peers { permanent_uuid: "a46d74955b29468281f1856f2db0863c" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 46803 } }
I20260926 15:16:25.517550 17774 raft_consensus.cc:374] T 230def51d8874b26aaaf211e361adbc2 P 30cad260cf9446968a17b19a2b3a139a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:16:25.517602 17775 ts_tablet_manager.cc:1428] T 230def51d8874b26aaaf211e361adbc2 P 9a94564df381458faad3c21d44ba4c39: Time spent starting tablet: real 0.018s user 0.009s sys 0.007s
I20260926 15:16:25.517676 17774 raft_consensus.cc:729] T 230def51d8874b26aaaf211e361adbc2 P 30cad260cf9446968a17b19a2b3a139a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30cad260cf9446968a17b19a2b3a139a, State: Initialized, Role: FOLLOWER
I20260926 15:16:25.518182 17488 heartbeater.cc:503] Master 127.6.209.254:35343 was elected leader, sending a full tablet report...
I20260926 15:16:25.518227 17774 consensus_queue.cc:260] T 230def51d8874b26aaaf211e361adbc2 P 30cad260cf9446968a17b19a2b3a139a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30cad260cf9446968a17b19a2b3a139a" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39345 } } peers { permanent_uuid: "9a94564df381458faad3c21d44ba4c39" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46489 } } peers { permanent_uuid: "a46d74955b29468281f1856f2db0863c" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 46803 } }
I20260926 15:16:25.520391 17622 heartbeater.cc:503] Master 127.6.209.254:35343 was elected leader, sending a full tablet report...
I20260926 15:16:25.521163 17773 ts_tablet_manager.cc:1428] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c: Time spent starting tablet: real 0.022s user 0.011s sys 0.010s
I20260926 15:16:25.522603 17755 heartbeater.cc:503] Master 127.6.209.254:35343 was elected leader, sending a full tablet report...
I20260926 15:16:25.524796 17774 ts_tablet_manager.cc:1428] T 230def51d8874b26aaaf211e361adbc2 P 30cad260cf9446968a17b19a2b3a139a: Time spent starting tablet: real 0.021s user 0.017s sys 0.000s
I20260926 15:16:25.548329 17779 raft_consensus.cc:482] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:16:25.548540 17779 raft_consensus.cc:504] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30cad260cf9446968a17b19a2b3a139a" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39345 } } peers { permanent_uuid: "9a94564df381458faad3c21d44ba4c39" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46489 } } peers { permanent_uuid: "a46d74955b29468281f1856f2db0863c" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 46803 } }
I20260926 15:16:25.549854 17779 leader_election.cc:288] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30cad260cf9446968a17b19a2b3a139a (127.6.209.195:39345), 9a94564df381458faad3c21d44ba4c39 (127.6.209.193:46489)
I20260926 15:16:25.555619 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:25.550068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:52879 (local address 127.6.209.195:39345)
0926 15:16:25.550340 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:25.550371 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:25.550965 (+ 594us) server_negotiation.cc:407] Connection header received
0926 15:16:25.551186 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:25.551197 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:25.551249 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:25.551341 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:16:25.551852 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.552554 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.554174 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.554846 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.555008 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:25.555072 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:25.555308 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 15:16:25.555405 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260926 15:16:25.555742 17782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:25.549924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:39345 (local address 127.6.209.194:52879)
0926 15:16:25.550868 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:25.550886 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:25.551054 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:25.551357 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:25.551362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:25.551386 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:16:25.551720 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:25.551728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.552667 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.552672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:25.554052 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:25.554061 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.555044 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.555049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:25.555178 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:25.555198 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 15:16:25.555287 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 15:16:25.555390 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":41,"thread_start_us":653,"threads_started":1}
I20260926 15:16:25.556435 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:25.550364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:35307 (local address 127.6.209.193:46489)
0926 15:16:25.550573 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:25.550582 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:25.551253 (+ 671us) server_negotiation.cc:407] Connection header received
0926 15:16:25.551465 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:25.551470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:25.551525 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:25.551626 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:16:25.552228 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.552932 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.554634 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.555442 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.555594 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:25.555666 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:25.556061 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 15:16:25.556167 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 15:16:25.556504 17783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:25.550220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:46489 (local address 127.6.209.194:35307)
0926 15:16:25.551144 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0926 15:16:25.551159 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:16:25.551321 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:16:25.551646 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:16:25.551651 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:16:25.551680 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:16:25.552074 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:25.552083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.553059 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.553065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:25.554472 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:16:25.554483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.555636 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.555641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:16:25.555901 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:25.555938 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 15:16:25.556039 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 15:16:25.556143 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":24,"thread_start_us":585,"threads_started":1}
I20260926 15:16:25.556571 17710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "230def51d8874b26aaaf211e361adbc2" candidate_uuid: "a46d74955b29468281f1856f2db0863c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30cad260cf9446968a17b19a2b3a139a" is_pre_election: true
I20260926 15:16:25.557006 17710 raft_consensus.cc:2454] T 230def51d8874b26aaaf211e361adbc2 P 30cad260cf9446968a17b19a2b3a139a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a46d74955b29468281f1856f2db0863c in term 0.
I20260926 15:16:25.557446 17443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "230def51d8874b26aaaf211e361adbc2" candidate_uuid: "a46d74955b29468281f1856f2db0863c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a94564df381458faad3c21d44ba4c39" is_pre_election: true
I20260926 15:16:25.557575 17513 leader_election.cc:302] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30cad260cf9446968a17b19a2b3a139a, a46d74955b29468281f1856f2db0863c; no voters:
I20260926 15:16:25.557919 17779 raft_consensus.cc:2781] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:16:25.557883 17443 raft_consensus.cc:2454] T 230def51d8874b26aaaf211e361adbc2 P 9a94564df381458faad3c21d44ba4c39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a46d74955b29468281f1856f2db0863c in term 0.
I20260926 15:16:25.558056 17779 raft_consensus.cc:482] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:16:25.558175 17779 raft_consensus.cc:3037] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:16:25.560132 17779 raft_consensus.cc:504] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30cad260cf9446968a17b19a2b3a139a" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39345 } } peers { permanent_uuid: "9a94564df381458faad3c21d44ba4c39" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46489 } } peers { permanent_uuid: "a46d74955b29468281f1856f2db0863c" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 46803 } }
I20260926 15:16:25.560676 17779 leader_election.cc:288] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c [CANDIDATE]: Term 1 election: Requested vote from peers 30cad260cf9446968a17b19a2b3a139a (127.6.209.195:39345), 9a94564df381458faad3c21d44ba4c39 (127.6.209.193:46489)
I20260926 15:16:25.561074 17710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "230def51d8874b26aaaf211e361adbc2" candidate_uuid: "a46d74955b29468281f1856f2db0863c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30cad260cf9446968a17b19a2b3a139a"
I20260926 15:16:25.561069 17443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "230def51d8874b26aaaf211e361adbc2" candidate_uuid: "a46d74955b29468281f1856f2db0863c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a94564df381458faad3c21d44ba4c39"
I20260926 15:16:25.561250 17710 raft_consensus.cc:3037] T 230def51d8874b26aaaf211e361adbc2 P 30cad260cf9446968a17b19a2b3a139a [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:16:25.561268 17443 raft_consensus.cc:3037] T 230def51d8874b26aaaf211e361adbc2 P 9a94564df381458faad3c21d44ba4c39 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:16:25.563246 17710 raft_consensus.cc:2454] T 230def51d8874b26aaaf211e361adbc2 P 30cad260cf9446968a17b19a2b3a139a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a46d74955b29468281f1856f2db0863c in term 1.
I20260926 15:16:25.563340 17443 raft_consensus.cc:2454] T 230def51d8874b26aaaf211e361adbc2 P 9a94564df381458faad3c21d44ba4c39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a46d74955b29468281f1856f2db0863c in term 1.
I20260926 15:16:25.563601 17513 leader_election.cc:302] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30cad260cf9446968a17b19a2b3a139a, a46d74955b29468281f1856f2db0863c; no voters:
I20260926 15:16:25.563901 17779 raft_consensus.cc:2781] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:16:25.564265 17779 raft_consensus.cc:686] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c [term 1 LEADER]: Becoming Leader. State: Replica: a46d74955b29468281f1856f2db0863c, State: Running, Role: LEADER
I20260926 15:16:25.564675 17779 consensus_queue.cc:237] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30cad260cf9446968a17b19a2b3a139a" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39345 } } peers { permanent_uuid: "9a94564df381458faad3c21d44ba4c39" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46489 } } peers { permanent_uuid: "a46d74955b29468281f1856f2db0863c" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 46803 } }
I20260926 15:16:25.568722 17297 catalog_manager.cc:5495] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c reported cstate change: term changed from 0 to 1, leader changed from <none> to a46d74955b29468281f1856f2db0863c (127.6.209.194). New cstate: current_term: 1 leader_uuid: "a46d74955b29468281f1856f2db0863c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30cad260cf9446968a17b19a2b3a139a" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a94564df381458faad3c21d44ba4c39" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a46d74955b29468281f1856f2db0863c" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 46803 } health_report { overall_health: HEALTHY } } }
I20260926 15:16:25.609879 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:25.605668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50526 (local address 127.6.209.194:46803)
0926 15:16:25.605827 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:25.605831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:25.605929 (+ 98us) server_negotiation.cc:407] Connection header received
0926 15:16:25.606069 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:25.606073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:25.606144 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:25.606279 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:16:25.606577 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.607179 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.608227 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.608562 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.608639 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:25.609064 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:16:25.609485 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:16:25.609580 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:25.609643 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 15:16:25.609696 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 15:16:25.616621 17443 raft_consensus.cc:1261] T 230def51d8874b26aaaf211e361adbc2 P 9a94564df381458faad3c21d44ba4c39 [term 1 FOLLOWER]: Refusing update from remote peer a46d74955b29468281f1856f2db0863c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:16:25.616624 17710 raft_consensus.cc:1261] T 230def51d8874b26aaaf211e361adbc2 P 30cad260cf9446968a17b19a2b3a139a [term 1 FOLLOWER]: Refusing update from remote peer a46d74955b29468281f1856f2db0863c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:16:25.617421 17784 consensus_queue.cc:1035] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a94564df381458faad3c21d44ba4c39" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:16:25.617671 17779 consensus_queue.cc:1035] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30cad260cf9446968a17b19a2b3a139a" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:16:25.628360 17787 mvcc.cc:205] Tried to move back new op lower bound from 7333624977877811200 to 7333624977676902400. Current Snapshot: MvccSnapshot[applied={T|T < 7333624977877811200}]
I20260926 15:16:25.641029 17297 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:16:25.641268 17297 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:16:25.644503 17297 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a27f70ee04874ec2afdd5ba57660758d: Sending DeleteTablet for 3 replicas of tablet 230def51d8874b26aaaf211e361adbc2
I20260926 15:16:25.645485 17690 tablet_service.cc:1514] Processing DeleteTablet for tablet 230def51d8874b26aaaf211e361adbc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:16:25 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:51706
I20260926 15:16:25.645635 17423 tablet_service.cc:1514] Processing DeleteTablet for tablet 230def51d8874b26aaaf211e361adbc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:16:25 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:36038
I20260926 15:16:25.646322 17557 tablet_service.cc:1514] Processing DeleteTablet for tablet 230def51d8874b26aaaf211e361adbc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:16:25 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:50520
I20260926 15:16:25.646737 17798 tablet_replica.cc:330] T 230def51d8874b26aaaf211e361adbc2 P 9a94564df381458faad3c21d44ba4c39: stopping tablet replica
I20260926 15:16:25.647047 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 17360
I20260926 15:16:25.647179 17798 raft_consensus.cc:2229] T 230def51d8874b26aaaf211e361adbc2 P 9a94564df381458faad3c21d44ba4c39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:16:25.647647 17798 raft_consensus.cc:2258] T 230def51d8874b26aaaf211e361adbc2 P 9a94564df381458faad3c21d44ba4c39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:16:25.656689 17798 ts_tablet_manager.cc:1905] T 230def51d8874b26aaaf211e361adbc2 P 9a94564df381458faad3c21d44ba4c39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:16:25.658799 17799 tablet_replica.cc:330] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c: stopping tablet replica
I20260926 15:16:25.659371 17799 raft_consensus.cc:2229] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:16:25.660122 17799 raft_consensus.cc:2258] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:16:25.660933 17797 tablet_replica.cc:330] T 230def51d8874b26aaaf211e361adbc2 P 30cad260cf9446968a17b19a2b3a139a: stopping tablet replica
I20260926 15:16:25.661299 17797 raft_consensus.cc:2229] T 230def51d8874b26aaaf211e361adbc2 P 30cad260cf9446968a17b19a2b3a139a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:16:25.661664 17797 raft_consensus.cc:2258] T 230def51d8874b26aaaf211e361adbc2 P 30cad260cf9446968a17b19a2b3a139a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:16:25.661929 17799 ts_tablet_manager.cc:1905] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:16:25.663640 17797 ts_tablet_manager.cc:1905] T 230def51d8874b26aaaf211e361adbc2 P 30cad260cf9446968a17b19a2b3a139a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:16:25.666971 17798 ts_tablet_manager.cc:1918] T 230def51d8874b26aaaf211e361adbc2 P 9a94564df381458faad3c21d44ba4c39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:16:25.667147 17798 log.cc:1196] T 230def51d8874b26aaaf211e361adbc2 P 9a94564df381458faad3c21d44ba4c39: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-0/wal/wals/230def51d8874b26aaaf211e361adbc2
I20260926 15:16:25.667626 17798 ts_tablet_manager.cc:1939] T 230def51d8874b26aaaf211e361adbc2 P 9a94564df381458faad3c21d44ba4c39: Deleting consensus metadata
I20260926 15:16:25.669181 17285 catalog_manager.cc:4841] TS 9a94564df381458faad3c21d44ba4c39 (127.6.209.193:46489): tablet 230def51d8874b26aaaf211e361adbc2 (table test-table [id=cf48701156b348aab0ee420e21ac1b3d]) successfully deleted
I20260926 15:16:25.669960 17799 ts_tablet_manager.cc:1918] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:16:25.669956 17797 ts_tablet_manager.cc:1918] T 230def51d8874b26aaaf211e361adbc2 P 30cad260cf9446968a17b19a2b3a139a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:16:25.670087 17797 log.cc:1196] T 230def51d8874b26aaaf211e361adbc2 P 30cad260cf9446968a17b19a2b3a139a: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/230def51d8874b26aaaf211e361adbc2
I20260926 15:16:25.670202 17799 log.cc:1196] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/230def51d8874b26aaaf211e361adbc2
I20260926 15:16:25.670609 17797 ts_tablet_manager.cc:1939] T 230def51d8874b26aaaf211e361adbc2 P 30cad260cf9446968a17b19a2b3a139a: Deleting consensus metadata
I20260926 15:16:25.670778 17799 ts_tablet_manager.cc:1939] T 230def51d8874b26aaaf211e361adbc2 P a46d74955b29468281f1856f2db0863c: Deleting consensus metadata
I20260926 15:16:25.672086 17286 catalog_manager.cc:4841] TS 30cad260cf9446968a17b19a2b3a139a (127.6.209.195:39345): tablet 230def51d8874b26aaaf211e361adbc2 (table test-table [id=cf48701156b348aab0ee420e21ac1b3d]) successfully deleted
I20260926 15:16:25.672252 17285 catalog_manager.cc:4841] TS a46d74955b29468281f1856f2db0863c (127.6.209.194:46803): tablet 230def51d8874b26aaaf211e361adbc2 (table test-table [id=cf48701156b348aab0ee420e21ac1b3d]) successfully deleted
Sep 26 15:16:25 dist-test-slave-j6q7 krb5kdc[17254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.193@KRBTEST.COM
I20260926 15:16:25.682902 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:25.648413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36044 (local address 127.6.209.193:46489)
0926 15:16:25.648564 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:25.648570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:25.663435 (+ 14865us) server_negotiation.cc:407] Connection header received
0926 15:16:25.663627 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:25.663633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:25.663715 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:25.663807 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:25.670079 (+ 6272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.671114 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.672706 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.673261 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.673373 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:25.676595 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:25.676606 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:25.676621 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:16:25.676656 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:25.680157 (+ 3501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:25.680774 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:25.680782 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:25.680786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:25.680954 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:25.681287 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:25.681292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:25.681296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:25.681717 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:25.682000 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:25.682366 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 15:16:25.682584 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 15:16:25.683535 17483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:16:25.765234 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 17360
I20260926 15:16:25.788177 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 17494
Sep 26 15:16:25 dist-test-slave-j6q7 krb5kdc[17254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.194@KRBTEST.COM
I20260926 15:16:25.802987 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:25.788889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50534 (local address 127.6.209.194:46803)
0926 15:16:25.789032 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:25.789036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:25.789051 (+ 15us) server_negotiation.cc:407] Connection header received
0926 15:16:25.789239 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:25.789243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:25.789317 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:25.789387 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:25.790764 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.791606 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.793410 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.793760 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.793836 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:25.797817 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:25.797827 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:25.797840 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:16:25.797870 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:25.800370 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:25.801110 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:25.801117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:25.801119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:25.801248 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:25.801612 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:25.801621 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:25.801623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:25.801951 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:25.802295 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:25.802499 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 15:16:25.802686 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 15:16:25.803540 17617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:16:25.914111 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 17494
I20260926 15:16:25.937040 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 17627
Sep 26 15:16:25 dist-test-slave-j6q7 krb5kdc[17254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.195@KRBTEST.COM
I20260926 15:16:25.952390 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:16:25.938029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51716 (local address 127.6.209.195:39345)
0926 15:16:25.938219 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 15:16:25.938223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:16:25.938238 (+ 15us) server_negotiation.cc:407] Connection header received
0926 15:16:25.938289 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:16:25.938292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:16:25.938347 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:16:25.938427 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:16:25.939993 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.940703 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.942679 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:16:25.943046 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:16:25.943136 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:16:25.947541 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:16:25.947553 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:16:25.947567 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:16:25.947593 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:16:25.949972 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:25.950539 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:25.950545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:25.950547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:25.950658 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:16:25.950986 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:16:25.950993 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:16:25.950997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:16:25.951340 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:16:25.951636 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 15:16:25.951892 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 15:16:25.952109 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 15:16:25.952996 17750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:16:26.035596 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 17627
I20260926 15:16:26.050522 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 17267
I20260926 15:16:26.051535 17329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:16:26.149061 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 17267
2026-09-26T15:16:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3769 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:16:26 dist-test-slave-j6q7 krb5kdc[17807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:16:26 dist-test-slave-j6q7 krb5kdc[17807](info): set up 2 sockets
Sep 26 15:16:26 dist-test-slave-j6q7 krb5kdc[17807](info): commencing operation
krb5kdc: starting...
W20260926 15:16:27.145423 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:16:27 dist-test-slave-j6q7 krb5kdc[17807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:16:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:16:27Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.209.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.209.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.209.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/rangeradmin_127.6.209.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.209.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/rangeradmin_127.6.209.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.209.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.209.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.209.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/rangerlookup_127.6.209.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.209.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/rangerlookup_127.6.209.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.209.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.209.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.209.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/HTTP_127.6.209.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.209.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/HTTP_127.6.209.212@KRBTEST.COM.keytab.
I20260926 15:16:27.388550 6983 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskFXn57Q/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/postgres -l logfile start
2026-09-26 15:16:29.542 UTC [17837] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 15:16:29.542 UTC [17837] LOG: listening on IPv4 address "127.6.209.212", port 59711
2026-09-26 15:16:29.545 UTC [17837] LOG: listening on Unix socket "/tmp/.s.PGSQL.59711"
2026-09-26 15:16:29.564 UTC [17840] LOG: database system was shut down at 2026-09-26 15:16:28 UTC
2026-09-26 15:16:29.567 UTC [17837] LOG: database system is ready to accept connections
I20260926 15:16:30.445185 6983 mini_postgres.cc:96] Postgres bound to 59711
2026-09-26 15:16:30.460 UTC [17848] FATAL: database "slave" does not exist
127.6.209.212:59711 - accepting connections
I20260926 15:16:30.460963 6983 mini_ranger.cc:161] Starting Ranger...
I20260926 15:16:30.480851 6983 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 15:16:30.897274 6983 mini_ranger.cc:87] Created ranger Postgres database
I20260926 15:16:30.897447 6983 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ranger-admin
2026-09-26 15:16:31,157 [I] DB FLAVOR :POSTGRES
2026-09-26 15:16:31,158 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 15:16:31,158 [I] Checking connection..
2026-09-26 15:16:31,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:59711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 15:16:31,385 [I] Checking connection passed.
2026-09-26 15:16:31,386 [I] --------- Verifying version history table ---------
2026-09-26 15:16:31,386 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:59711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 15:16:31,620 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 15:16:31,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:59711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 15:16:31,849 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 15:16:31,849 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 15:16:31,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:59711/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 15:16:32.077 UTC [17912] WARNING: there is no transaction in progress
2026-09-26 15:16:32,082 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 15:16:32,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:59711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 15:16:32,283 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 15:16:32,283 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 15:16:32,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:59711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 15:16:32,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:59711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 15:16:32,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:59711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j6q7.c.gcp-upstream.internal','N') ;"
2026-09-26 15:16:32,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:59711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 15:16:33,163 [I] Table x_portal_user does not exist in database ranger
2026-09-26 15:16:33,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:59711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 15:16:33,385 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 15:16:33,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:59711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 15:16:33,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:59711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 15:16:33,845 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 15:16:33,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:59711/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 15:16:34.667 UTC [18020] WARNING: there is no transaction in progress
2026-09-26 15:16:34.681 UTC [18020] WARNING: there is no transaction in progress
2026-09-26 15:16:34.695 UTC [18020] WARNING: there is no transaction in progress
2026-09-26 15:16:34.709 UTC [18020] WARNING: there is no transaction in progress
2026-09-26 15:16:34.722 UTC [18020] WARNING: there is no transaction in progress
2026-09-26 15:16:34.738 UTC [18020] WARNING: there is no transaction in progress
2026-09-26 15:16:34.784 UTC [18020] WARNING: there is no transaction in progress
2026-09-26 15:16:34.790 UTC [18020] WARNING: there is no transaction in progress
2026-09-26 15:16:34.802 UTC [18020] WARNING: there is no transaction in progress
2026-09-26 15:16:34.810 UTC [18020] WARNING: there is no transaction in progress
2026-09-26 15:16:34.819 UTC [18020] WARNING: there is no transaction in progress
2026-09-26 15:16:34.830 UTC [18020] WARNING: there is no transaction in progress
2026-09-26 15:16:34.838 UTC [18020] WARNING: there is no transaction in progress
2026-09-26 15:16:34.843 UTC [18020] WARNING: there is no transaction in progress
2026-09-26 15:16:34.848 UTC [18020] WARNING: there is no transaction in progress
2026-09-26 15:16:34.853 UTC [18020] WARNING: there is no transaction in progress
2026-09-26 15:16:35,397 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 15:16:35,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:59711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 15:16:35,632 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 15:16:35,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:59711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 15:16:35,847 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 15:16:35,847 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:59711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 15:16:36,075 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 15:16:36,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:59711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 15:16:36,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:59711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 15:16:36,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:59711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j6q7.c.gcp-upstream.internal';"
2026-09-26 15:16:36,698 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 15:16:36,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:59711/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 15:16:36,917 [I] DB_PATCHES have already been applied
I20260926 15:16:36.923311 6983 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ranger-admin:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskFXn57Q/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 15:16:36.923441 6983 mini_ranger.cc:193] Using host: 127.6.209.212
I20260926 15:16:36.936299 6983 mini_ranger.cc:227] Ranger admin URL: http://127.6.209.212:37391
Sep 26, 2026 3:16:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:16:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:16:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 3:16:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 3:16:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 3:16:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 3:16:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 15:16:37.959748 6983 mini_ranger.cc:160] Time spent starting Ranger: real 7.499s user 0.005s sys 0.024s
Sep 26, 2026 3:16:55 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 3:16:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 26, 2026 3:16:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 3:16:55 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 3:16:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 3:16:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 3:16:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 3:16:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 3:16:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 3:16:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 3:16:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 3:16:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 3:16:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 3:16:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 3:16:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 3:16:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 3:16:55 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 3:16:55 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 15:16:56.474987 6983 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.209.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.209.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.209.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/rangerkms_127.6.209.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.209.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/rangerkms_127.6.209.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.209.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.209.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.209.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/HTTP_127.6.209.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.209.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/HTTP_127.6.209.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 15:16:56 dist-test-slave-j6q7 krb5kdc[17807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435816, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 15:16:56.632347 6983 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 15:16:56.652030 6983 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 15:16:57.397269 6983 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 15:16:57.397362 6983 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ranger-kms
I20260926 15:16:57.397505 6983 mini_ranger_kms.cc:227] Using postgres at 127.6.209.212:59711
2026-09-26 15:16:57,759 [I] DB FLAVOR :POSTGRES
2026-09-26 15:16:57,760 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 15:16:57,760 [I] Checking connection
2026-09-26 15:16:57,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:59711/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 15:16:57,976 [I] connection success
2026-09-26 15:16:57,977 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 15:16:57,977 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 15:16:57,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:59711/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 15:16:58,177 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 15:16:58,177 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 15:16:58,177 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 15:16:58,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:59711/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 15:16:58,408 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 15:16:58.842211 6983 mini_ranger_kms.cc:310] Created kms service
I20260926 15:16:59.000939 6983 mini_ranger_kms.cc:326] Created kudu user
I20260926 15:16:59.041465 6983 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 15:16:59 dist-test-slave-j6q7 krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435816, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.209.212@KRBTEST.COM
I20260926 15:16:59.371646 6983 mini_ranger_kms.cc:384] Added ranger policy
I20260926 15:16:59.371917 6983 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ranger-kms:/tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 15:16:59.371965 6983 mini_ranger_kms.cc:240] Using host: 127.6.209.212
I20260926 15:16:59.393041 6983 mini_ranger_kms.cc:276] Ranger KMS PID: 18200
I20260926 15:16:59.393196 6983 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.209.212:49427
Sep 26, 2026 3:17:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:17:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:17:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 3:17:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 3:17:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 3:17:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 15:17:01 dist-test-slave-j6q7 krb5kdc[17807](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790435821, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.209.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:17:01.465121 6983 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.833s user 0.125s sys 0.248s
I20260926 15:17:01.465381 6983 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 15:17:01.465592 6983 mini_ranger_kms.cc:401] 127.6.209.212:49427/kms/v1/keys
Sep 26 15:17:01 dist-test-slave-j6q7 krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435821, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.209.212@KRBTEST.COM for HTTP/127.6.209.212@KRBTEST.COM
Sep 26 15:17:02 dist-test-slave-j6q7 krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435821, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.209.212@KRBTEST.COM for HTTP/127.6.209.212@KRBTEST.COM
Sep 26 15:17:04 dist-test-slave-j6q7 krb5kdc[17807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:17:04.209276 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:37859
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:37859
--ranger_config_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.209.212:49427/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:17:04.530933 18250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:17:04.531172 18250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:17:04.531334 18250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:17:04.538098 18250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:17:04.538187 18250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:17:04.538229 18250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:17:04.538307 18250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:17:04.538374 18250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:17:04.547154 18250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.209.212:49427/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:37859
--ranger_config_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:37859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:17:04.548192 18250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:17:04.549465 18250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:17:04.560936 18257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:04.561261 18256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:04.561345 18259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:17:04.561993 18250 server_base.cc:1034] running on GCE node
I20260926 15:17:04.562651 18250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:17:04.563813 18250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:17:04.564997 18250 hybrid_clock.cc:648] HybridClock initialized: now 1790435824564956 us; error 28 us; skew 500 ppm
I20260926 15:17:04.565361 18250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:17:04 dist-test-slave-j6q7 krb5kdc[17807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435824, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:17:04.572691 18250 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:17:04.573938 18250 webserver.cc:460] Webserver started at http://127.6.209.254:39185/ using document root <none> and password file <none>
I20260926 15:17:04.574481 18250 fs_manager.cc:359] Metadata directory not provided
I20260926 15:17:04.574601 18250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:17:04.574862 18250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 15:17:04 dist-test-slave-j6q7 krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435824, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for HTTP/127.6.209.212@KRBTEST.COM
I20260926 15:17:04.635596 18250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "db5118f8392d43f98438611049631948"
format_stamp: "Formatted at 2026-09-26 15:17:04 on dist-test-slave-j6q7"
server_key: "297461bb4d70b1ad845425bb398b686d"
server_key_iv: "10e8ca8b26e9c2954874b736c09ca542"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:04.636565 18250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "db5118f8392d43f98438611049631948"
format_stamp: "Formatted at 2026-09-26 15:17:04 on dist-test-slave-j6q7"
server_key: "297461bb4d70b1ad845425bb398b686d"
server_key_iv: "10e8ca8b26e9c2954874b736c09ca542"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:04.647148 18250 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.005s sys 0.004s
I20260926 15:17:04.665367 18266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:04.666998 18250 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 15:17:04.667150 18250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "db5118f8392d43f98438611049631948"
format_stamp: "Formatted at 2026-09-26 15:17:04 on dist-test-slave-j6q7"
server_key: "297461bb4d70b1ad845425bb398b686d"
server_key_iv: "10e8ca8b26e9c2954874b736c09ca542"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:04.667348 18250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:17:04.704774 18250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:17:04.709040 18250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:17:04.709340 18250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:17:04.728751 18250 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:37859
I20260926 15:17:04.728741 18317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:37859 every 8 connection(s)
I20260926 15:17:04.730511 18250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:17:04.733915 18318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:17:04.736240 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 18250
I20260926 15:17:04.736416 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:17:04.737255 6983 external_mini_cluster.cc:1428] Setting key 035e4b91675a9b87ae7e0f9113a14247
I20260926 15:17:04.747165 18318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db5118f8392d43f98438611049631948: Bootstrap starting.
I20260926 15:17:04.751821 18318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db5118f8392d43f98438611049631948: Neither blocks nor log segments found. Creating new log.
I20260926 15:17:04.753060 18318 log.cc:824] T 00000000000000000000000000000000 P db5118f8392d43f98438611049631948: Log is configured to *not* fsync() on all Append() calls
I20260926 15:17:04.755475 18318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db5118f8392d43f98438611049631948: No bootstrap required, opened a new log
Sep 26 15:17:04 dist-test-slave-j6q7 krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:17:04.764262 18318 raft_consensus.cc:348] T 00000000000000000000000000000000 P db5118f8392d43f98438611049631948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db5118f8392d43f98438611049631948" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37859 } }
I20260926 15:17:04.764657 18318 raft_consensus.cc:374] T 00000000000000000000000000000000 P db5118f8392d43f98438611049631948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:17:04.764747 18318 raft_consensus.cc:729] T 00000000000000000000000000000000 P db5118f8392d43f98438611049631948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db5118f8392d43f98438611049631948, State: Initialized, Role: FOLLOWER
I20260926 15:17:04.765381 18318 consensus_queue.cc:260] T 00000000000000000000000000000000 P db5118f8392d43f98438611049631948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db5118f8392d43f98438611049631948" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37859 } }
I20260926 15:17:04.765672 18318 raft_consensus.cc:388] T 00000000000000000000000000000000 P db5118f8392d43f98438611049631948 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:17:04.765806 18318 raft_consensus.cc:482] T 00000000000000000000000000000000 P db5118f8392d43f98438611049631948 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:17:04.765987 18318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db5118f8392d43f98438611049631948 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:17:04.768613 18318 raft_consensus.cc:504] T 00000000000000000000000000000000 P db5118f8392d43f98438611049631948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db5118f8392d43f98438611049631948" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37859 } }
I20260926 15:17:04.769165 18318 leader_election.cc:302] T 00000000000000000000000000000000 P db5118f8392d43f98438611049631948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db5118f8392d43f98438611049631948; no voters:
I20260926 15:17:04.769744 18318 leader_election.cc:288] T 00000000000000000000000000000000 P db5118f8392d43f98438611049631948 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:17:04.769877 18323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db5118f8392d43f98438611049631948 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:17:04.771615 18323 raft_consensus.cc:686] T 00000000000000000000000000000000 P db5118f8392d43f98438611049631948 [term 1 LEADER]: Becoming Leader. State: Replica: db5118f8392d43f98438611049631948, State: Running, Role: LEADER
I20260926 15:17:04.772594 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:04.739223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55492 (local address 127.6.209.254:37859)
0926 15:17:04.752554 (+ 13331us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:04.752570 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:04.752894 (+ 324us) server_negotiation.cc:407] Connection header received
0926 15:17:04.755009 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:04.755047 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:04.755392 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:04.755749 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:04.757196 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:04.758311 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:04.759848 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:04.760591 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:04.760718 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:04.764436 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:04.764444 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:04.764459 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:04.764499 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:04.767934 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:04.768490 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:04.768499 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:04.768507 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:04.768677 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:04.769068 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:04.769072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:04.769075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:04.769705 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:04.769996 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:04.770555 (+ 559us) server_negotiation.cc:299] Negotiation successful
0926 15:17:04.771172 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13101,"thread_start_us":12891,"threads_started":1}
I20260926 15:17:04.772984 18318 sys_catalog.cc:564] T 00000000000000000000000000000000 P db5118f8392d43f98438611049631948 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:17:04.773034 18323 consensus_queue.cc:237] T 00000000000000000000000000000000 P db5118f8392d43f98438611049631948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db5118f8392d43f98438611049631948" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37859 } }
I20260926 15:17:04.778251 18324 sys_catalog.cc:455] T 00000000000000000000000000000000 P db5118f8392d43f98438611049631948 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db5118f8392d43f98438611049631948" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db5118f8392d43f98438611049631948" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37859 } } }
I20260926 15:17:04.778609 18324 sys_catalog.cc:458] T 00000000000000000000000000000000 P db5118f8392d43f98438611049631948 [sys.catalog]: This master's current role is: LEADER
I20260926 15:17:04.778939 18325 sys_catalog.cc:455] T 00000000000000000000000000000000 P db5118f8392d43f98438611049631948 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db5118f8392d43f98438611049631948. Latest consensus state: current_term: 1 leader_uuid: "db5118f8392d43f98438611049631948" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db5118f8392d43f98438611049631948" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37859 } } }
I20260926 15:17:04.779946 18325 sys_catalog.cc:458] T 00000000000000000000000000000000 P db5118f8392d43f98438611049631948 [sys.catalog]: This master's current role is: LEADER
I20260926 15:17:04.783573 18332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:17:04.788215 18332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:17:04.788738 18318 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 15:17:04.806734 18332 catalog_manager.cc:1347] Generated new cluster ID: f3b0ac180f6043b6bfc9e61766a7628b
I20260926 15:17:04.806936 18332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:17:04.849517 18332 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:17:04.851164 18332 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:17:04.864208 18332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db5118f8392d43f98438611049631948: Generated new TSK 0
I20260926 15:17:04.865535 18332 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
I20260926 15:17:05.508102 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.209.212:49427/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:37859
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:17:05 dist-test-slave-j6q7 krb5kdc[17807](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790435825, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 15:17:05.867003 18363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:17:05.867255 18363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:17:05.867384 18363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:17:05.873792 18363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:17:05.873907 18363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:17:05.874063 18363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:17:05.883148 18363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.209.212:49427/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:37859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:17:05.884297 18363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:17:05.885566 18363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:17:05.901517 18373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:05.901556 18374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:05.903859 18376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:17:05.904412 18363 server_base.cc:1034] running on GCE node
I20260926 15:17:05.905035 18363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:17:05.905956 18363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:17:05.907169 18363 hybrid_clock.cc:648] HybridClock initialized: now 1790435825907081 us; error 85 us; skew 500 ppm
I20260926 15:17:05.907440 18363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:17:05 dist-test-slave-j6q7 krb5kdc[17807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435825, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:17:05.910266 18363 init.cc:377] Logged in from keytab as kudu/127.6.209.193@KRBTEST.COM (short username kudu)
I20260926 15:17:05.913908 18363 webserver.cc:460] Webserver started at http://127.6.209.193:33151/ using document root <none> and password file <none>
I20260926 15:17:05.914535 18363 fs_manager.cc:359] Metadata directory not provided
I20260926 15:17:05.914664 18363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:17:05.914942 18363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 15:17:05 dist-test-slave-j6q7 krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435825, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for HTTP/127.6.209.212@KRBTEST.COM
I20260926 15:17:05.930649 18363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "46d98b410ab740aab4ac5bdf4a711136"
format_stamp: "Formatted at 2026-09-26 15:17:05 on dist-test-slave-j6q7"
server_key: "b19965a029f3e2adbe87657ced2c0d0a"
server_key_iv: "118508d04301c4989acea1a201a798b1"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:05.931903 18363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "46d98b410ab740aab4ac5bdf4a711136"
format_stamp: "Formatted at 2026-09-26 15:17:05 on dist-test-slave-j6q7"
server_key: "b19965a029f3e2adbe87657ced2c0d0a"
server_key_iv: "118508d04301c4989acea1a201a798b1"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:05.943236 18363 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.004s sys 0.006s
I20260926 15:17:05.973794 18383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:05.977223 18363 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.000s sys 0.013s
I20260926 15:17:05.977458 18363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "46d98b410ab740aab4ac5bdf4a711136"
format_stamp: "Formatted at 2026-09-26 15:17:05 on dist-test-slave-j6q7"
server_key: "b19965a029f3e2adbe87657ced2c0d0a"
server_key_iv: "118508d04301c4989acea1a201a798b1"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:05.977775 18363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:17:06.037565 18363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:17:06.041507 18363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:17:06.041795 18363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:17:06.042611 18363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:17:06.043829 18363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:17:06.043928 18363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:06.044034 18363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:17:06.044093 18363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:06.156909 18363 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:34833
I20260926 15:17:06.157059 18495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:34833 every 8 connection(s)
I20260926 15:17:06.158293 18363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:17:06.164072 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 18363
I20260926 15:17:06.164278 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:17:06.165282 6983 external_mini_cluster.cc:1428] Setting key 9bb34f8a03d9c88794ad4f56c7062720
Sep 26 15:17:06 dist-test-slave-j6q7 krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435825, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:17:06.204698 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:06.161792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:37859 (local address 127.6.209.193:41439)
0926 15:17:06.173844 (+ 12052us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:06.173906 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:06.176429 (+ 2523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:06.185019 (+ 8590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:06.185031 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:06.185795 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:17:06.186695 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:06.186720 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:06.189832 (+ 3112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:06.189838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:06.192229 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:06.192244 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:06.193551 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:06.193560 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:06.193883 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:06.194934 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:17:06.194964 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:17:06.197495 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:17:06.200845 (+ 3350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:06.200853 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:06.200868 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:06.201343 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:06.201719 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:06.201723 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:06.201726 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:06.201938 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:06.202518 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:17:06.202525 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:17:06.202909 (+ 384us) client_negotiation.cc:769] Sending connection context
0926 15:17:06.203141 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 15:17:06.203554 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10951,"thread_start_us":11023,"threads_started":1}
I20260926 15:17:06.205511 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:06.161335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:41439 (local address 127.6.209.254:37859)
0926 15:17:06.184276 (+ 22941us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:06.184283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:06.184303 (+ 20us) server_negotiation.cc:407] Connection header received
0926 15:17:06.184366 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:06.184372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:06.184461 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:06.186905 (+ 2444us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:06.187083 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:06.188105 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:06.192437 (+ 4332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:06.193095 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:06.193229 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:06.197720 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:06.197725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:06.197729 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:06.197765 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:06.200663 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:06.201466 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:06.201471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:06.201474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:06.201609 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:06.202044 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:06.202048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:06.202051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:06.202338 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:06.202451 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:06.204986 (+ 2535us) server_negotiation.cc:299] Negotiation successful
0926 15:17:06.205212 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22816,"thread_start_us":23193,"threads_started":1}
WARNING: no policy specified for kudu/127.6.209.194@KRBTEST.COM; defaulting to no policy
I20260926 15:17:06.207803 18496 heartbeater.cc:347] Connected to a master server at 127.6.209.254:37859
I20260926 15:17:06.208158 18496 heartbeater.cc:464] Registering TS with master...
I20260926 15:17:06.209241 18496 heartbeater.cc:511] Master 127.6.209.254:37859 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.194@KRBTEST.COM" created.
I20260926 15:17:06.211613 18283 ts_manager.cc:194] Registered new tserver with Master: 46d98b410ab740aab4ac5bdf4a711136 (127.6.209.193:34833)
I20260926 15:17:06.213315 18283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.193@KRBTEST.COM'} at 127.6.209.193:41439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
I20260926 15:17:06.242013 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.209.212:49427/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:37859
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:17:06 dist-test-slave-j6q7 krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435825, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for HTTP/127.6.209.212@KRBTEST.COM
Sep 26 15:17:06 dist-test-slave-j6q7 krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435825, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for HTTP/127.6.209.212@KRBTEST.COM
W20260926 15:17:06.684892 18502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:17:06.685357 18502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:17:06.685573 18502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:17:06.694659 18502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:17:06.694813 18502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:17:06.695000 18502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:17:06.709163 18502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.209.212:49427/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:37859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:17:06.710840 18502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:17:06.712387 18502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:17:06.757637 18509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:06.763582 18510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:06.769287 18512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:17:06.773106 18502 server_base.cc:1034] running on GCE node
I20260926 15:17:06.774068 18502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:17:06.776254 18502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:17:06.777504 18502 hybrid_clock.cc:648] HybridClock initialized: now 1790435826777413 us; error 99 us; skew 500 ppm
I20260926 15:17:06.777892 18502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:17:06 dist-test-slave-j6q7 krb5kdc[17807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435826, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:17:06.781880 18502 init.cc:377] Logged in from keytab as kudu/127.6.209.194@KRBTEST.COM (short username kudu)
I20260926 15:17:06.786836 18502 webserver.cc:460] Webserver started at http://127.6.209.194:35135/ using document root <none> and password file <none>
I20260926 15:17:06.787492 18502 fs_manager.cc:359] Metadata directory not provided
I20260926 15:17:06.787609 18502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:17:06.787925 18502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 15:17:06 dist-test-slave-j6q7 krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435826, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for HTTP/127.6.209.212@KRBTEST.COM
I20260926 15:17:06.803185 18502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "36f73fa5861e40ab804973347bd8a210"
format_stamp: "Formatted at 2026-09-26 15:17:06 on dist-test-slave-j6q7"
server_key: "154f501034397d1efa65755a04168315"
server_key_iv: "03143c22edc0e36882d742b4cf372c43"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:06.804133 18502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "36f73fa5861e40ab804973347bd8a210"
format_stamp: "Formatted at 2026-09-26 15:17:06 on dist-test-slave-j6q7"
server_key: "154f501034397d1efa65755a04168315"
server_key_iv: "03143c22edc0e36882d742b4cf372c43"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:06.829247 18502 fs_manager.cc:691] Time spent creating directory manager: real 0.025s user 0.016s sys 0.008s
I20260926 15:17:06.852247 18526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:06.853937 18502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:17:06.854120 18502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "36f73fa5861e40ab804973347bd8a210"
format_stamp: "Formatted at 2026-09-26 15:17:06 on dist-test-slave-j6q7"
server_key: "154f501034397d1efa65755a04168315"
server_key_iv: "03143c22edc0e36882d742b4cf372c43"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:06.854372 18502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:17:06.879019 18502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:17:06.884069 18502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:17:06.884433 18502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:17:06.885655 18502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:17:06.888060 18502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:17:06.888170 18502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:06.888309 18502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:17:06.888396 18502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:06.927838 18502 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:33897
I20260926 15:17:06.927892 18638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:33897 every 8 connection(s)
I20260926 15:17:06.929324 18502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:17:06.931008 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 18502
I20260926 15:17:06.931165 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:17:06.931691 6983 external_mini_cluster.cc:1428] Setting key 3f657a3a1e135734d04f5f702e3ca93f
Sep 26 15:17:06 dist-test-slave-j6q7 krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435826, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.209.195@KRBTEST.COM; defaulting to no policy
I20260926 15:17:06.981349 18641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:06.933074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:53817 (local address 127.6.209.254:37859)
0926 15:17:06.960173 (+ 27099us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:06.960180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:06.960202 (+ 22us) server_negotiation.cc:407] Connection header received
0926 15:17:06.963073 (+ 2871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:06.963080 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:06.963189 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:06.963300 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:06.966450 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:06.967478 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:06.969631 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:06.970213 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:06.970313 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:06.974243 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:06.974249 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:06.974253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:06.974285 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:06.977827 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:06.978760 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:06.978765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:06.978769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:06.978892 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:06.979417 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:06.979422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:06.979425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:06.979747 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:06.979857 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:06.980749 (+ 892us) server_negotiation.cc:299] Negotiation successful
0926 15:17:06.980971 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26975,"thread_start_us":27152,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.195@KRBTEST.COM" created.
I20260926 15:17:06.983280 18642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:06.932650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:37859 (local address 127.6.209.194:53817)
0926 15:17:06.956197 (+ 23547us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:06.956250 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:06.958664 (+ 2414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:06.963743 (+ 5079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:06.963755 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:06.964393 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:17:06.965224 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:06.965250 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:06.967652 (+ 2402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:06.967658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:06.969480 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:06.969493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:06.970581 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:06.970587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:06.970824 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:06.971548 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:17:06.971572 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:17:06.974038 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:17:06.978074 (+ 4036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:06.978083 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:06.978098 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:06.978632 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:06.979026 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:06.979030 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:06.979033 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:06.979305 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:06.979942 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:17:06.979949 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:17:06.980341 (+ 392us) client_negotiation.cc:769] Sending connection context
0926 15:17:06.981566 (+ 1225us) client_negotiation.cc:240] Negotiation successful
0926 15:17:06.982012 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23207,"thread_start_us":26364,"threads_started":1}
I20260926 15:17:06.991338 18639 heartbeater.cc:347] Connected to a master server at 127.6.209.254:37859
I20260926 15:17:06.991701 18639 heartbeater.cc:464] Registering TS with master...
I20260926 15:17:06.992543 18639 heartbeater.cc:511] Master 127.6.209.254:37859 requested a full tablet report, sending...
I20260926 15:17:06.994190 18283 ts_manager.cc:194] Registered new tserver with Master: 36f73fa5861e40ab804973347bd8a210 (127.6.209.194:33897)
I20260926 15:17:06.995301 18283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.194@KRBTEST.COM'} at 127.6.209.194:53817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
I20260926 15:17:07.019855 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.209.212:49427/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:37859
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:17:07.221382 18496 heartbeater.cc:503] Master 127.6.209.254:37859 was elected leader, sending a full tablet report...
W20260926 15:17:07.372105 18645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:17:07.372339 18645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:17:07.372473 18645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:17:07.379304 18645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:17:07.379417 18645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:17:07.379555 18645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:17:07.388053 18645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.209.212:49427/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:37859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:17:07.389114 18645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:17:07.390141 18645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:17:07.401613 18653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:07.401636 18650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:07.401584 18651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:17:07.401929 18645 server_base.cc:1034] running on GCE node
I20260926 15:17:07.402612 18645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:17:07.403335 18645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:17:07.404502 18645 hybrid_clock.cc:648] HybridClock initialized: now 1790435827404464 us; error 32 us; skew 500 ppm
I20260926 15:17:07.404712 18645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:17:07 dist-test-slave-j6q7 krb5kdc[17807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435827, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:17:07.407459 18645 init.cc:377] Logged in from keytab as kudu/127.6.209.195@KRBTEST.COM (short username kudu)
I20260926 15:17:07.408636 18645 webserver.cc:460] Webserver started at http://127.6.209.195:43915/ using document root <none> and password file <none>
I20260926 15:17:07.409235 18645 fs_manager.cc:359] Metadata directory not provided
I20260926 15:17:07.409363 18645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:17:07.409631 18645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 15:17:07 dist-test-slave-j6q7 krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435827, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for HTTP/127.6.209.212@KRBTEST.COM
I20260926 15:17:07.423545 18645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "8bf33586d08e454d8c10c7d733e8bd28"
format_stamp: "Formatted at 2026-09-26 15:17:07 on dist-test-slave-j6q7"
server_key: "9ff4e48fa02bdc653523c780725a1aea"
server_key_iv: "1790e9c4f63f7202685ee8f925acbcb6"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:07.424556 18645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "8bf33586d08e454d8c10c7d733e8bd28"
format_stamp: "Formatted at 2026-09-26 15:17:07 on dist-test-slave-j6q7"
server_key: "9ff4e48fa02bdc653523c780725a1aea"
server_key_iv: "1790e9c4f63f7202685ee8f925acbcb6"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:07.429878 18645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 15:17:07.443998 18660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:07.445289 18645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:17:07.445452 18645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "8bf33586d08e454d8c10c7d733e8bd28"
format_stamp: "Formatted at 2026-09-26 15:17:07 on dist-test-slave-j6q7"
server_key: "9ff4e48fa02bdc653523c780725a1aea"
server_key_iv: "1790e9c4f63f7202685ee8f925acbcb6"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:07.445654 18645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:17:07.462301 18645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:17:07.465648 18645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:17:07.465914 18645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:17:07.467105 18645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:17:07.468159 18645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:17:07.468243 18645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:07.468362 18645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:17:07.468420 18645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:07.499577 18645 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:36609
I20260926 15:17:07.499617 18772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:36609 every 8 connection(s)
I20260926 15:17:07.500731 18645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:17:07.504626 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 18645
I20260926 15:17:07.504778 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:17:07.505295 6983 external_mini_cluster.cc:1428] Setting key b5decea58a01f64f1f09edaa587030c0
Sep 26 15:17:07 dist-test-slave-j6q7 krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435827, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:17:07.517995 18775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:07.502928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:41409 (local address 127.6.209.254:37859)
0926 15:17:07.503348 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:07.503355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:07.504181 (+ 826us) server_negotiation.cc:407] Connection header received
0926 15:17:07.505991 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:07.505997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:07.506079 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:07.506202 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:07.507873 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.508649 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.510367 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.510706 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.510789 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:07.513520 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:07.513524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:07.513527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:07.513560 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:07.515780 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:07.516392 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:07.516395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:07.516397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:07.516517 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:07.516870 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:07.516873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:07.516875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:07.517085 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:07.517184 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:07.517632 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 15:17:07.517780 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":177,"threads_started":1}
I20260926 15:17:07.518832 18776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:07.503433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:37859 (local address 127.6.209.195:41409)
0926 15:17:07.504046 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:07.504079 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:07.505669 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:07.506463 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:07.506470 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:07.507025 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:17:07.507707 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:07.507730 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.508768 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.508772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:07.510248 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:07.510258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.510857 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.510861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:07.511028 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:07.511580 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:17:07.511598 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:17:07.513353 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:17:07.515902 (+ 2549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:07.515908 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:07.515917 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:07.516284 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:07.516594 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:07.516596 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:07.516599 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:07.516755 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:07.517170 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:17:07.517175 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:17:07.517394 (+ 219us) client_negotiation.cc:769] Sending connection context
0926 15:17:07.517562 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 15:17:07.517863 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":247,"threads_started":1}
I20260926 15:17:07.519959 18773 heartbeater.cc:347] Connected to a master server at 127.6.209.254:37859
I20260926 15:17:07.520206 18773 heartbeater.cc:464] Registering TS with master...
I20260926 15:17:07.520776 18773 heartbeater.cc:511] Master 127.6.209.254:37859 requested a full tablet report, sending...
I20260926 15:17:07.521909 18283 ts_manager.cc:194] Registered new tserver with Master: 8bf33586d08e454d8c10c7d733e8bd28 (127.6.209.195:36609)
I20260926 15:17:07.522584 18283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.195@KRBTEST.COM'} at 127.6.209.195:41409
I20260926 15:17:07.532006 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:17:07.546535 18775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:07.535860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55508 (local address 127.6.209.254:37859)
0926 15:17:07.535984 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:07.535988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:07.536423 (+ 435us) server_negotiation.cc:407] Connection header received
0926 15:17:07.536531 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:07.536534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:07.536591 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:07.536661 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:07.538235 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.538832 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.539836 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.540183 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.540292 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:07.542482 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:07.542486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:07.542489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:07.542513 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:07.544511 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:07.545046 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:07.545049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:07.545050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:07.545155 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:07.545470 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:07.545473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:07.545475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:07.545686 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:07.545847 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:07.546080 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 15:17:07.546226 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 15:17:07.549650 18283 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:17:07.555789 18283 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:17:07.580044 18787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:07.567049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:33897 (local address 127.0.0.1:50966)
0926 15:17:07.568957 (+ 1908us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:07.568971 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:07.569057 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:07.569957 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:07.569964 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:07.569993 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:17:07.570486 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:07.570499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.572680 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.572686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:07.576314 (+ 3628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:07.576334 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.578895 (+ 2561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.578903 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:07.579140 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:07.579202 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 15:17:07.579345 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 15:17:07.579550 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1766,"thread_start_us":1608,"threads_started":1}
I20260926 15:17:07.581106 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:07.567801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.6.209.194:33897)
0926 15:17:07.568234 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:07.568241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:07.569179 (+ 938us) server_negotiation.cc:407] Connection header received
0926 15:17:07.569268 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:07.569274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:07.569486 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:07.569682 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:17:07.570708 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.572499 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.576596 (+ 4097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.578349 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.578490 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:07.578706 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:07.580435 (+ 1729us) server_negotiation.cc:299] Negotiation successful
0926 15:17:07.580659 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"spinlock_wait_cycles":16256,"thread_start_us":289,"threads_started":1}
I20260926 15:17:07.581933 18786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:07.566710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:36609 (local address 127.0.0.1:43682)
0926 15:17:07.567318 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:07.567350 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:07.567490 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:07.568345 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:07.568351 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:07.568377 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:17:07.568876 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:07.568888 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.572508 (+ 3620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.572516 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:07.577181 (+ 4665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:07.577198 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.580957 (+ 3759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.580966 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:07.581213 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:07.581247 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 15:17:07.581341 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 15:17:07.581514 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":426,"threads_started":1}
I20260926 15:17:07.582762 18788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:07.567395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43682 (local address 127.6.209.195:36609)
0926 15:17:07.567934 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:07.567940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:07.567962 (+ 22us) server_negotiation.cc:407] Connection header received
0926 15:17:07.568021 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:07.568025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:07.568177 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:07.568337 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:17:07.569247 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.572316 (+ 3069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.577379 (+ 5063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.580346 (+ 2967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.580496 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:07.580751 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:07.582195 (+ 1444us) server_negotiation.cc:299] Negotiation successful
0926 15:17:07.582389 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":548,"threads_started":1}
I20260926 15:17:07.586134 18574 tablet_service.cc:1467] Processing CreateTablet for tablet b3812fe16ade4b1aa6d1271623bf6790 (DEFAULT_TABLE table=test-table [id=78df49404a894f07a57a09b7cea23a26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:17:07.586619 18708 tablet_service.cc:1467] Processing CreateTablet for tablet b3812fe16ade4b1aa6d1271623bf6790 (DEFAULT_TABLE table=test-table [id=78df49404a894f07a57a09b7cea23a26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:17:07.587427 18574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3812fe16ade4b1aa6d1271623bf6790. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:17:07.587757 18708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3812fe16ade4b1aa6d1271623bf6790. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:17:07.590312 18787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:07.569085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:34833 (local address 127.0.0.1:49666)
0926 15:17:07.581690 (+ 12605us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:07.581710 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:07.581813 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:07.582854 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:07.582858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:07.582879 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:17:07.583034 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:07.583045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.585831 (+ 2786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.585835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:07.587456 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:07.587477 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.589690 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.589696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:07.589845 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:07.589873 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 15:17:07.589963 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 15:17:07.590085 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12483,"thread_start_us":13787,"threads_started":1}
I20260926 15:17:07.590446 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:07.569305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49666 (local address 127.6.209.193:34833)
0926 15:17:07.582439 (+ 13134us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:07.582447 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:07.582470 (+ 23us) server_negotiation.cc:407] Connection header received
0926 15:17:07.582546 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:07.582552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:07.582702 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:07.582941 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:17:07.583167 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.585696 (+ 2529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.587623 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.589276 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.589399 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:07.589581 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:07.590019 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 15:17:07.590172 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12975,"thread_start_us":12990,"threads_started":1}
I20260926 15:17:07.593510 18431 tablet_service.cc:1467] Processing CreateTablet for tablet b3812fe16ade4b1aa6d1271623bf6790 (DEFAULT_TABLE table=test-table [id=78df49404a894f07a57a09b7cea23a26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:17:07.594652 18431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3812fe16ade4b1aa6d1271623bf6790. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:17:07.598227 18792 tablet_bootstrap.cc:492] T b3812fe16ade4b1aa6d1271623bf6790 P 36f73fa5861e40ab804973347bd8a210: Bootstrap starting.
I20260926 15:17:07.600224 18793 tablet_bootstrap.cc:492] T b3812fe16ade4b1aa6d1271623bf6790 P 8bf33586d08e454d8c10c7d733e8bd28: Bootstrap starting.
I20260926 15:17:07.601613 18792 tablet_bootstrap.cc:654] T b3812fe16ade4b1aa6d1271623bf6790 P 36f73fa5861e40ab804973347bd8a210: Neither blocks nor log segments found. Creating new log.
I20260926 15:17:07.603102 18792 log.cc:824] T b3812fe16ade4b1aa6d1271623bf6790 P 36f73fa5861e40ab804973347bd8a210: Log is configured to *not* fsync() on all Append() calls
I20260926 15:17:07.603816 18793 tablet_bootstrap.cc:654] T b3812fe16ade4b1aa6d1271623bf6790 P 8bf33586d08e454d8c10c7d733e8bd28: Neither blocks nor log segments found. Creating new log.
I20260926 15:17:07.604941 18793 log.cc:824] T b3812fe16ade4b1aa6d1271623bf6790 P 8bf33586d08e454d8c10c7d733e8bd28: Log is configured to *not* fsync() on all Append() calls
I20260926 15:17:07.605962 18792 tablet_bootstrap.cc:492] T b3812fe16ade4b1aa6d1271623bf6790 P 36f73fa5861e40ab804973347bd8a210: No bootstrap required, opened a new log
I20260926 15:17:07.606243 18792 ts_tablet_manager.cc:1397] T b3812fe16ade4b1aa6d1271623bf6790 P 36f73fa5861e40ab804973347bd8a210: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 15:17:07.607133 18796 tablet_bootstrap.cc:492] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136: Bootstrap starting.
I20260926 15:17:07.608119 18793 tablet_bootstrap.cc:492] T b3812fe16ade4b1aa6d1271623bf6790 P 8bf33586d08e454d8c10c7d733e8bd28: No bootstrap required, opened a new log
I20260926 15:17:07.608349 18793 ts_tablet_manager.cc:1397] T b3812fe16ade4b1aa6d1271623bf6790 P 8bf33586d08e454d8c10c7d733e8bd28: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 15:17:07.610677 18796 tablet_bootstrap.cc:654] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136: Neither blocks nor log segments found. Creating new log.
I20260926 15:17:07.611647 18796 log.cc:824] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136: Log is configured to *not* fsync() on all Append() calls
I20260926 15:17:07.613231 18792 raft_consensus.cc:348] T b3812fe16ade4b1aa6d1271623bf6790 P 36f73fa5861e40ab804973347bd8a210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf33586d08e454d8c10c7d733e8bd28" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36609 } } peers { permanent_uuid: "36f73fa5861e40ab804973347bd8a210" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33897 } } peers { permanent_uuid: "46d98b410ab740aab4ac5bdf4a711136" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 34833 } }
I20260926 15:17:07.613530 18792 raft_consensus.cc:374] T b3812fe16ade4b1aa6d1271623bf6790 P 36f73fa5861e40ab804973347bd8a210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:17:07.613636 18792 raft_consensus.cc:729] T b3812fe16ade4b1aa6d1271623bf6790 P 36f73fa5861e40ab804973347bd8a210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36f73fa5861e40ab804973347bd8a210, State: Initialized, Role: FOLLOWER
I20260926 15:17:07.614195 18792 consensus_queue.cc:260] T b3812fe16ade4b1aa6d1271623bf6790 P 36f73fa5861e40ab804973347bd8a210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf33586d08e454d8c10c7d733e8bd28" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36609 } } peers { permanent_uuid: "36f73fa5861e40ab804973347bd8a210" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33897 } } peers { permanent_uuid: "46d98b410ab740aab4ac5bdf4a711136" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 34833 } }
I20260926 15:17:07.615094 18639 heartbeater.cc:503] Master 127.6.209.254:37859 was elected leader, sending a full tablet report...
I20260926 15:17:07.615453 18796 tablet_bootstrap.cc:492] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136: No bootstrap required, opened a new log
I20260926 15:17:07.615692 18796 ts_tablet_manager.cc:1397] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260926 15:17:07.615725 18792 ts_tablet_manager.cc:1428] T b3812fe16ade4b1aa6d1271623bf6790 P 36f73fa5861e40ab804973347bd8a210: Time spent starting tablet: real 0.009s user 0.006s sys 0.004s
I20260926 15:17:07.617671 18793 raft_consensus.cc:348] T b3812fe16ade4b1aa6d1271623bf6790 P 8bf33586d08e454d8c10c7d733e8bd28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf33586d08e454d8c10c7d733e8bd28" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36609 } } peers { permanent_uuid: "36f73fa5861e40ab804973347bd8a210" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33897 } } peers { permanent_uuid: "46d98b410ab740aab4ac5bdf4a711136" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 34833 } }
I20260926 15:17:07.618018 18793 raft_consensus.cc:374] T b3812fe16ade4b1aa6d1271623bf6790 P 8bf33586d08e454d8c10c7d733e8bd28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:17:07.618096 18793 raft_consensus.cc:729] T b3812fe16ade4b1aa6d1271623bf6790 P 8bf33586d08e454d8c10c7d733e8bd28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf33586d08e454d8c10c7d733e8bd28, State: Initialized, Role: FOLLOWER
I20260926 15:17:07.618572 18793 consensus_queue.cc:260] T b3812fe16ade4b1aa6d1271623bf6790 P 8bf33586d08e454d8c10c7d733e8bd28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf33586d08e454d8c10c7d733e8bd28" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36609 } } peers { permanent_uuid: "36f73fa5861e40ab804973347bd8a210" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33897 } } peers { permanent_uuid: "46d98b410ab740aab4ac5bdf4a711136" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 34833 } }
I20260926 15:17:07.619477 18773 heartbeater.cc:503] Master 127.6.209.254:37859 was elected leader, sending a full tablet report...
I20260926 15:17:07.621105 18793 ts_tablet_manager.cc:1428] T b3812fe16ade4b1aa6d1271623bf6790 P 8bf33586d08e454d8c10c7d733e8bd28: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260926 15:17:07.624167 18796 raft_consensus.cc:348] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf33586d08e454d8c10c7d733e8bd28" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36609 } } peers { permanent_uuid: "36f73fa5861e40ab804973347bd8a210" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33897 } } peers { permanent_uuid: "46d98b410ab740aab4ac5bdf4a711136" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 34833 } }
I20260926 15:17:07.624405 18796 raft_consensus.cc:374] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:17:07.624464 18796 raft_consensus.cc:729] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46d98b410ab740aab4ac5bdf4a711136, State: Initialized, Role: FOLLOWER
I20260926 15:17:07.624848 18796 consensus_queue.cc:260] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf33586d08e454d8c10c7d733e8bd28" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36609 } } peers { permanent_uuid: "36f73fa5861e40ab804973347bd8a210" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33897 } } peers { permanent_uuid: "46d98b410ab740aab4ac5bdf4a711136" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 34833 } }
I20260926 15:17:07.625907 18796 ts_tablet_manager.cc:1428] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136: Time spent starting tablet: real 0.010s user 0.005s sys 0.005s
W20260926 15:17:07.695873 18497 tablet.cc:2369] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:17:07.716961 18640 tablet.cc:2369] T b3812fe16ade4b1aa6d1271623bf6790 P 36f73fa5861e40ab804973347bd8a210: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:17:07.752146 18774 tablet.cc:2369] T b3812fe16ade4b1aa6d1271623bf6790 P 8bf33586d08e454d8c10c7d733e8bd28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:17:07.813812 18800 raft_consensus.cc:482] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:17:07.814020 18800 raft_consensus.cc:504] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf33586d08e454d8c10c7d733e8bd28" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36609 } } peers { permanent_uuid: "36f73fa5861e40ab804973347bd8a210" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33897 } } peers { permanent_uuid: "46d98b410ab740aab4ac5bdf4a711136" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 34833 } }
I20260926 15:17:07.815460 18800 leader_election.cc:288] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bf33586d08e454d8c10c7d733e8bd28 (127.6.209.195:36609), 36f73fa5861e40ab804973347bd8a210 (127.6.209.194:33897)
I20260926 15:17:07.821511 18788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:07.815973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:51055 (local address 127.6.209.195:36609)
0926 15:17:07.816147 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:07.816154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:07.816923 (+ 769us) server_negotiation.cc:407] Connection header received
0926 15:17:07.817124 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:07.817129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:07.817180 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:07.817259 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:17:07.817708 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.818304 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.819901 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.820582 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.820750 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:07.820844 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:07.821137 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 15:17:07.821251 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 15:17:07.821592 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:07.815868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:46737 (local address 127.6.209.194:33897)
0926 15:17:07.816074 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:07.816081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:07.817259 (+ 1178us) server_negotiation.cc:407] Connection header received
0926 15:17:07.817418 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:07.817425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:07.817473 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:07.817562 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:17:07.817983 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.818597 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.820054 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.820720 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.820888 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:07.820954 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:07.821235 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 15:17:07.821349 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 15:17:07.821652 18801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:07.815768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:36609 (local address 127.6.209.193:51055)
0926 15:17:07.816798 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:07.816854 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:07.816997 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:07.817280 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:07.817283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:07.817306 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:17:07.817582 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:07.817590 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.818440 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.818446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:07.819775 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:07.819786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.820766 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.820772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:07.820944 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:07.821001 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 15:17:07.821116 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 15:17:07.821245 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"mutex_wait_us":30,"thread_start_us":793,"threads_started":1}
I20260926 15:17:07.821725 18802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:07.815858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:33897 (local address 127.6.209.193:46737)
0926 15:17:07.817171 (+ 1313us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:07.817183 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:07.817308 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:07.817584 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:07.817588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:07.817607 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:17:07.817864 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:07.817871 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.818728 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.818733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:07.819932 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:07.819942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.820926 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.820932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:07.821078 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:07.821116 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 15:17:07.821212 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 15:17:07.821324 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1226,"mutex_wait_us":2,"thread_start_us":1037,"threads_started":1}
I20260926 15:17:07.822582 18728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3812fe16ade4b1aa6d1271623bf6790" candidate_uuid: "46d98b410ab740aab4ac5bdf4a711136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf33586d08e454d8c10c7d733e8bd28" is_pre_election: true
I20260926 15:17:07.822567 18594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3812fe16ade4b1aa6d1271623bf6790" candidate_uuid: "46d98b410ab740aab4ac5bdf4a711136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36f73fa5861e40ab804973347bd8a210" is_pre_election: true
I20260926 15:17:07.823019 18728 raft_consensus.cc:2454] T b3812fe16ade4b1aa6d1271623bf6790 P 8bf33586d08e454d8c10c7d733e8bd28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46d98b410ab740aab4ac5bdf4a711136 in term 0.
I20260926 15:17:07.823431 18594 raft_consensus.cc:2454] T b3812fe16ade4b1aa6d1271623bf6790 P 36f73fa5861e40ab804973347bd8a210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46d98b410ab740aab4ac5bdf4a711136 in term 0.
I20260926 15:17:07.824009 18386 leader_election.cc:302] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46d98b410ab740aab4ac5bdf4a711136, 8bf33586d08e454d8c10c7d733e8bd28; no voters:
I20260926 15:17:07.824375 18800 raft_consensus.cc:2781] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:17:07.824506 18800 raft_consensus.cc:482] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:17:07.824591 18800 raft_consensus.cc:3037] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:17:07.826689 18800 raft_consensus.cc:504] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf33586d08e454d8c10c7d733e8bd28" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36609 } } peers { permanent_uuid: "36f73fa5861e40ab804973347bd8a210" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33897 } } peers { permanent_uuid: "46d98b410ab740aab4ac5bdf4a711136" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 34833 } }
I20260926 15:17:07.827147 18800 leader_election.cc:288] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136 [CANDIDATE]: Term 1 election: Requested vote from peers 8bf33586d08e454d8c10c7d733e8bd28 (127.6.209.195:36609), 36f73fa5861e40ab804973347bd8a210 (127.6.209.194:33897)
I20260926 15:17:07.827497 18728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3812fe16ade4b1aa6d1271623bf6790" candidate_uuid: "46d98b410ab740aab4ac5bdf4a711136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf33586d08e454d8c10c7d733e8bd28"
I20260926 15:17:07.827500 18594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3812fe16ade4b1aa6d1271623bf6790" candidate_uuid: "46d98b410ab740aab4ac5bdf4a711136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36f73fa5861e40ab804973347bd8a210"
I20260926 15:17:07.827657 18728 raft_consensus.cc:3037] T b3812fe16ade4b1aa6d1271623bf6790 P 8bf33586d08e454d8c10c7d733e8bd28 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:17:07.827664 18594 raft_consensus.cc:3037] T b3812fe16ade4b1aa6d1271623bf6790 P 36f73fa5861e40ab804973347bd8a210 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:17:07.829452 18728 raft_consensus.cc:2454] T b3812fe16ade4b1aa6d1271623bf6790 P 8bf33586d08e454d8c10c7d733e8bd28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46d98b410ab740aab4ac5bdf4a711136 in term 1.
I20260926 15:17:07.829612 18594 raft_consensus.cc:2454] T b3812fe16ade4b1aa6d1271623bf6790 P 36f73fa5861e40ab804973347bd8a210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46d98b410ab740aab4ac5bdf4a711136 in term 1.
I20260926 15:17:07.829818 18386 leader_election.cc:302] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46d98b410ab740aab4ac5bdf4a711136, 8bf33586d08e454d8c10c7d733e8bd28; no voters:
I20260926 15:17:07.830052 18800 raft_consensus.cc:2781] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:17:07.830372 18800 raft_consensus.cc:686] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136 [term 1 LEADER]: Becoming Leader. State: Replica: 46d98b410ab740aab4ac5bdf4a711136, State: Running, Role: LEADER
I20260926 15:17:07.830739 18800 consensus_queue.cc:237] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf33586d08e454d8c10c7d733e8bd28" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36609 } } peers { permanent_uuid: "36f73fa5861e40ab804973347bd8a210" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33897 } } peers { permanent_uuid: "46d98b410ab740aab4ac5bdf4a711136" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 34833 } }
I20260926 15:17:07.834666 18283 catalog_manager.cc:5495] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46d98b410ab740aab4ac5bdf4a711136 (127.6.209.193). New cstate: current_term: 1 leader_uuid: "46d98b410ab740aab4ac5bdf4a711136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf33586d08e454d8c10c7d733e8bd28" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36f73fa5861e40ab804973347bd8a210" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46d98b410ab740aab4ac5bdf4a711136" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 34833 } health_report { overall_health: HEALTHY } } }
I20260926 15:17:07.864629 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:07.859484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49682 (local address 127.6.209.193:34833)
0926 15:17:07.859632 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:07.859637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:07.859706 (+ 69us) server_negotiation.cc:407] Connection header received
0926 15:17:07.859895 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:07.859899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:07.859967 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:07.860084 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:17:07.860455 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.861133 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.862737 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.863066 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.863182 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:07.863592 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:17:07.864050 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:17:07.864256 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:07.864304 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 15:17:07.864403 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 15:17:07.871125 18594 raft_consensus.cc:1261] T b3812fe16ade4b1aa6d1271623bf6790 P 36f73fa5861e40ab804973347bd8a210 [term 1 FOLLOWER]: Refusing update from remote peer 46d98b410ab740aab4ac5bdf4a711136: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:17:07.871188 18728 raft_consensus.cc:1261] T b3812fe16ade4b1aa6d1271623bf6790 P 8bf33586d08e454d8c10c7d733e8bd28 [term 1 FOLLOWER]: Refusing update from remote peer 46d98b410ab740aab4ac5bdf4a711136: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:17:07.871874 18803 consensus_queue.cc:1035] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36f73fa5861e40ab804973347bd8a210" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:17:07.872159 18800 consensus_queue.cc:1035] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bf33586d08e454d8c10c7d733e8bd28" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:17:07.880972 18806 mvcc.cc:205] Tried to move back new op lower bound from 7333625150952718336 to 7333625150797467648. Current Snapshot: MvccSnapshot[applied={T|T < 7333625150952718336}]
I20260926 15:17:07.893488 18281 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:17:07.893725 18281 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:17:07.896757 18281 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db5118f8392d43f98438611049631948: Sending DeleteTablet for 3 replicas of tablet b3812fe16ade4b1aa6d1271623bf6790
I20260926 15:17:07.897611 18708 tablet_service.cc:1514] Processing DeleteTablet for tablet b3812fe16ade4b1aa6d1271623bf6790 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:17:07 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:43682
I20260926 15:17:07.898015 18431 tablet_service.cc:1514] Processing DeleteTablet for tablet b3812fe16ade4b1aa6d1271623bf6790 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:17:07 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:49666
I20260926 15:17:07.898170 18574 tablet_service.cc:1514] Processing DeleteTablet for tablet b3812fe16ade4b1aa6d1271623bf6790 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:17:07 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:50966
I20260926 15:17:07.899129 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 18363
I20260926 15:17:07.921826 18817 tablet_replica.cc:330] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136: stopping tablet replica
Sep 26 15:17:07 dist-test-slave-j6q7 krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.193@KRBTEST.COM
I20260926 15:17:07.922343 18817 raft_consensus.cc:2229] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:17:07.922500 18818 tablet_replica.cc:330] T b3812fe16ade4b1aa6d1271623bf6790 P 36f73fa5861e40ab804973347bd8a210: stopping tablet replica
I20260926 15:17:07.922868 18818 raft_consensus.cc:2229] T b3812fe16ade4b1aa6d1271623bf6790 P 36f73fa5861e40ab804973347bd8a210 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:17:07.923301 18818 raft_consensus.cc:2258] T b3812fe16ade4b1aa6d1271623bf6790 P 36f73fa5861e40ab804973347bd8a210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:17:07.923803 18817 raft_consensus.cc:2258] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:17:07.923991 18816 tablet_replica.cc:330] T b3812fe16ade4b1aa6d1271623bf6790 P 8bf33586d08e454d8c10c7d733e8bd28: stopping tablet replica
I20260926 15:17:07.924331 18816 raft_consensus.cc:2229] T b3812fe16ade4b1aa6d1271623bf6790 P 8bf33586d08e454d8c10c7d733e8bd28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:17:07.924748 18816 raft_consensus.cc:2258] T b3812fe16ade4b1aa6d1271623bf6790 P 8bf33586d08e454d8c10c7d733e8bd28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:17:07.926697 18817 ts_tablet_manager.cc:1905] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:17:07.926687 18816 ts_tablet_manager.cc:1905] T b3812fe16ade4b1aa6d1271623bf6790 P 8bf33586d08e454d8c10c7d733e8bd28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:17:07.926697 18818 ts_tablet_manager.cc:1905] T b3812fe16ade4b1aa6d1271623bf6790 P 36f73fa5861e40ab804973347bd8a210: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:17:07.932785 18816 ts_tablet_manager.cc:1918] T b3812fe16ade4b1aa6d1271623bf6790 P 8bf33586d08e454d8c10c7d733e8bd28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:17:07.933000 18816 log.cc:1196] T b3812fe16ade4b1aa6d1271623bf6790 P 8bf33586d08e454d8c10c7d733e8bd28: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/b3812fe16ade4b1aa6d1271623bf6790
I20260926 15:17:07.933559 18816 ts_tablet_manager.cc:1939] T b3812fe16ade4b1aa6d1271623bf6790 P 8bf33586d08e454d8c10c7d733e8bd28: Deleting consensus metadata
I20260926 15:17:07.933678 18817 ts_tablet_manager.cc:1918] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:17:07.933887 18817 log.cc:1196] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-0/wal/wals/b3812fe16ade4b1aa6d1271623bf6790
I20260926 15:17:07.934497 18817 ts_tablet_manager.cc:1939] T b3812fe16ade4b1aa6d1271623bf6790 P 46d98b410ab740aab4ac5bdf4a711136: Deleting consensus metadata
I20260926 15:17:07.935681 18269 catalog_manager.cc:4841] TS 8bf33586d08e454d8c10c7d733e8bd28 (127.6.209.195:36609): tablet b3812fe16ade4b1aa6d1271623bf6790 (table test-table [id=78df49404a894f07a57a09b7cea23a26]) successfully deleted
I20260926 15:17:07.936429 18268 catalog_manager.cc:4841] TS 46d98b410ab740aab4ac5bdf4a711136 (127.6.209.193:34833): tablet b3812fe16ade4b1aa6d1271623bf6790 (table test-table [id=78df49404a894f07a57a09b7cea23a26]) successfully deleted
I20260926 15:17:07.938511 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:07.900490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49696 (local address 127.6.209.193:34833)
0926 15:17:07.900691 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:07.900701 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:07.910601 (+ 9900us) server_negotiation.cc:407] Connection header received
0926 15:17:07.910970 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:07.910977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:07.911054 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:07.911124 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:07.912961 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.914345 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.916705 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:07.917491 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:07.917598 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:07.924551 (+ 6953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:07.924565 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:07.924581 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:07.924632 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:07.928446 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:07.935998 (+ 7552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:07.936007 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:07.936011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:07.936187 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:07.936684 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:07.936689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:07.936692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:07.937170 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:07.937438 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:07.937967 (+ 529us) server_negotiation.cc:299] Negotiation successful
0926 15:17:07.938178 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 15:17:07.939680 18491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:17:07.945182 18818 ts_tablet_manager.cc:1918] T b3812fe16ade4b1aa6d1271623bf6790 P 36f73fa5861e40ab804973347bd8a210: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:17:07.945391 18818 log.cc:1196] T b3812fe16ade4b1aa6d1271623bf6790 P 36f73fa5861e40ab804973347bd8a210: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/b3812fe16ade4b1aa6d1271623bf6790
I20260926 15:17:07.946254 18818 ts_tablet_manager.cc:1939] T b3812fe16ade4b1aa6d1271623bf6790 P 36f73fa5861e40ab804973347bd8a210: Deleting consensus metadata
I20260926 15:17:07.948094 18268 catalog_manager.cc:4841] TS 36f73fa5861e40ab804973347bd8a210 (127.6.209.194:33897): tablet b3812fe16ade4b1aa6d1271623bf6790 (table test-table [id=78df49404a894f07a57a09b7cea23a26]) successfully deleted
I20260926 15:17:08.032078 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 18363
I20260926 15:17:08.048544 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 18502
Sep 26 15:17:08 dist-test-slave-j6q7 krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.194@KRBTEST.COM
I20260926 15:17:08.062605 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:08.049208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50974 (local address 127.6.209.194:33897)
0926 15:17:08.049375 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:08.049379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:08.049395 (+ 16us) server_negotiation.cc:407] Connection header received
0926 15:17:08.049531 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:08.049537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:08.049603 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:08.049704 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:08.050945 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:08.051600 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:08.052962 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:08.053278 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:08.053358 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:08.057729 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:08.057739 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:08.057753 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:08.057782 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:08.060244 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:08.061030 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:08.061036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:08.061038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:08.061155 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:08.061479 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:08.061482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:08.061483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:08.061772 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:08.061942 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:08.062170 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 15:17:08.062290 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 15:17:08.063158 18634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:17:08.142326 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 18502
I20260926 15:17:08.157572 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 18645
Sep 26 15:17:08 dist-test-slave-j6q7 krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.195@KRBTEST.COM
I20260926 15:17:08.171078 18788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:08.158234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43696 (local address 127.6.209.195:36609)
0926 15:17:08.158391 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:08.158396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:08.158475 (+ 79us) server_negotiation.cc:407] Connection header received
0926 15:17:08.158667 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:08.158671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:08.158726 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:08.158812 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:08.159925 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:08.160625 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:08.161933 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:08.162330 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:08.162447 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:08.166515 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:08.166527 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:08.166542 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:08.166571 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:08.168742 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:08.169372 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:08.169378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:08.169380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:08.169492 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:08.169804 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:08.169810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:08.169813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:08.170101 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:08.170367 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:08.170628 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 15:17:08.170798 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 15:17:08.171671 18768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:17:08.252748 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 18645
I20260926 15:17:08.267372 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 18250
I20260926 15:17:08.268327 18313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:17:08.365334 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 18250
I20260926 15:17:08.384102 6983 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 15:17:08.719054 6983 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 15:17:08.719266 6983 mini_ranger.cc:66] Stopping Ranger...
I20260926 15:17:09.055243 6983 mini_ranger.cc:68] Stopped Ranger
2026-09-26 15:17:09.055 UTC [17837] LOG: received smart shutdown request
2026-09-26 15:17:09.059 UTC [17837] LOG: background worker "logical replication launcher" (PID 17846) exited with exit code 1
2026-09-26 15:17:09.059 UTC [17841] LOG: shutting down
2026-09-26 15:17:09.199 UTC [17837] LOG: database system is shut down
2026-09-26T15:17:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (43161 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:17:09 dist-test-slave-j6q7 krb5kdc[18832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:17:09 dist-test-slave-j6q7 krb5kdc[18832](info): set up 2 sockets
Sep 26 15:17:09 dist-test-slave-j6q7 krb5kdc[18832](info): commencing operation
krb5kdc: starting...
W20260926 15:17:10.312697 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:17:10 dist-test-slave-j6q7 krb5kdc[18832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:17:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:17:10Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.209.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.209.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.209.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/rangeradmin_127.6.209.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.209.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/rangeradmin_127.6.209.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.209.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.209.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.209.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/rangerlookup_127.6.209.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.209.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/rangerlookup_127.6.209.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.209.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.209.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.209.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/HTTP_127.6.209.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.209.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/HTTP_127.6.209.212@KRBTEST.COM.keytab.
I20260926 15:17:10.563175 6983 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskFXn57Q/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/postgres -l logfile start
2026-09-26 15:17:12.695 UTC [18862] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 15:17:12.695 UTC [18862] LOG: listening on IPv4 address "127.6.209.212", port 55067
2026-09-26 15:17:12.698 UTC [18862] LOG: listening on Unix socket "/tmp/.s.PGSQL.55067"
2026-09-26 15:17:12.716 UTC [18865] LOG: database system was shut down at 2026-09-26 15:17:11 UTC
2026-09-26 15:17:12.719 UTC [18862] LOG: database system is ready to accept connections
I20260926 15:17:13.624524 6983 mini_postgres.cc:96] Postgres bound to 55067
2026-09-26 15:17:13.640 UTC [18873] FATAL: database "slave" does not exist
127.6.209.212:55067 - accepting connections
I20260926 15:17:13.640797 6983 mini_ranger.cc:161] Starting Ranger...
I20260926 15:17:13.663501 6983 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 15:17:14.072724 6983 mini_ranger.cc:87] Created ranger Postgres database
I20260926 15:17:14.072865 6983 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ranger-admin
2026-09-26 15:17:14,330 [I] DB FLAVOR :POSTGRES
2026-09-26 15:17:14,330 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 15:17:14,331 [I] Checking connection..
2026-09-26 15:17:14,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:55067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 15:17:14,569 [I] Checking connection passed.
2026-09-26 15:17:14,569 [I] --------- Verifying version history table ---------
2026-09-26 15:17:14,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:55067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 15:17:14,797 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 15:17:14,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:55067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 15:17:15,012 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 15:17:15,012 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 15:17:15,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:55067/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 15:17:15.258 UTC [18937] WARNING: there is no transaction in progress
2026-09-26 15:17:15,264 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 15:17:15,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:55067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 15:17:15,495 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 15:17:15,495 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 15:17:15,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:55067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 15:17:15,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:55067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 15:17:15,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:55067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j6q7.c.gcp-upstream.internal','N') ;"
2026-09-26 15:17:16,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:55067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 15:17:16,359 [I] Table x_portal_user does not exist in database ranger
2026-09-26 15:17:16,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:55067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 15:17:16,569 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 15:17:16,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:55067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 15:17:16,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:55067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 15:17:17,015 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 15:17:17,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:55067/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 15:17:17.854 UTC [19045] WARNING: there is no transaction in progress
2026-09-26 15:17:17.868 UTC [19045] WARNING: there is no transaction in progress
2026-09-26 15:17:17.882 UTC [19045] WARNING: there is no transaction in progress
2026-09-26 15:17:17.895 UTC [19045] WARNING: there is no transaction in progress
2026-09-26 15:17:17.911 UTC [19045] WARNING: there is no transaction in progress
2026-09-26 15:17:17.932 UTC [19045] WARNING: there is no transaction in progress
2026-09-26 15:17:17.977 UTC [19045] WARNING: there is no transaction in progress
2026-09-26 15:17:17.983 UTC [19045] WARNING: there is no transaction in progress
2026-09-26 15:17:17.995 UTC [19045] WARNING: there is no transaction in progress
2026-09-26 15:17:18.004 UTC [19045] WARNING: there is no transaction in progress
2026-09-26 15:17:18.013 UTC [19045] WARNING: there is no transaction in progress
2026-09-26 15:17:18.025 UTC [19045] WARNING: there is no transaction in progress
2026-09-26 15:17:18.033 UTC [19045] WARNING: there is no transaction in progress
2026-09-26 15:17:18.039 UTC [19045] WARNING: there is no transaction in progress
2026-09-26 15:17:18.044 UTC [19045] WARNING: there is no transaction in progress
2026-09-26 15:17:18.049 UTC [19045] WARNING: there is no transaction in progress
2026-09-26 15:17:18,593 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 15:17:18,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:55067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 15:17:18,833 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 15:17:18,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:55067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 15:17:19,038 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 15:17:19,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:55067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 15:17:19,240 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 15:17:19,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:55067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 15:17:19,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:55067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 15:17:19,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:55067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j6q7.c.gcp-upstream.internal';"
2026-09-26 15:17:19,895 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 15:17:19,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:55067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 15:17:20,119 [I] DB_PATCHES have already been applied
I20260926 15:17:20.124919 6983 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ranger-admin:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskFXn57Q/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 15:17:20.125021 6983 mini_ranger.cc:193] Using host: 127.6.209.212
I20260926 15:17:20.140141 6983 mini_ranger.cc:227] Ranger admin URL: http://127.6.209.212:38123
Sep 26, 2026 3:17:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:17:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:17:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 3:17:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 3:17:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 3:17:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 3:17:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 15:17:21.173888 6983 mini_ranger.cc:160] Time spent starting Ranger: real 7.533s user 0.005s sys 0.028s
Sep 26, 2026 3:17:38 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 3:17:38 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 3:17:38 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 3:17:38 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 3:17:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 3:17:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 3:17:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 3:17:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 3:17:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 3:17:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 3:17:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 3:17:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 3:17:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 3:17:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 3:17:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 3:17:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 3:17:38 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 3:17:39 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 15:17:39.866590 6983 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.209.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.209.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.209.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/rangerkms_127.6.209.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.209.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/rangerkms_127.6.209.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.209.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.209.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.209.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/HTTP_127.6.209.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.209.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/HTTP_127.6.209.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 15:17:40 dist-test-slave-j6q7 krb5kdc[18832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435860, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 15:17:40.028438 6983 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 15:17:40.046909 6983 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 15:17:40.803889 6983 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 15:17:40.803990 6983 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ranger-kms
I20260926 15:17:40.804040 6983 mini_ranger_kms.cc:227] Using postgres at 127.6.209.212:55067
2026-09-26 15:17:41,127 [I] DB FLAVOR :POSTGRES
2026-09-26 15:17:41,127 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 15:17:41,127 [I] Checking connection
2026-09-26 15:17:41,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:55067/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 15:17:41,333 [I] connection success
2026-09-26 15:17:41,333 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 15:17:41,333 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 15:17:41,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:55067/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 15:17:41,563 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 15:17:41,563 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 15:17:41,563 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 15:17:41,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.209.212:55067/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 15:17:41,819 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 15:17:42.211047 6983 mini_ranger_kms.cc:310] Created kms service
I20260926 15:17:42.373039 6983 mini_ranger_kms.cc:326] Created kudu user
I20260926 15:17:42.410904 6983 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 15:17:42 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435860, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.209.212@KRBTEST.COM
I20260926 15:17:42.750133 6983 mini_ranger_kms.cc:384] Added ranger policy
I20260926 15:17:42.750401 6983 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ranger-kms:/tmp/dist-test-taskFXn57Q/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskFXn57Q/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 15:17:42.750437 6983 mini_ranger_kms.cc:240] Using host: 127.6.209.212
I20260926 15:17:42.767167 6983 mini_ranger_kms.cc:276] Ranger KMS PID: 19225
I20260926 15:17:42.767273 6983 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.209.212:54783
Sep 26, 2026 3:17:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:17:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:17:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 3:17:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 3:17:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 3:17:44 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 15:17:44.832935 6983 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.804s user 0.087s sys 0.244s
I20260926 15:17:44.833204 6983 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 15:17:44.833391 6983 mini_ranger_kms.cc:401] 127.6.209.212:54783/kms/v1/keys
Sep 26 15:17:44 dist-test-slave-j6q7 krb5kdc[18832](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790435864, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.209.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 15:17:45 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435864, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.209.212@KRBTEST.COM for HTTP/127.6.209.212@KRBTEST.COM
Sep 26 15:17:45 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435864, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.209.212@KRBTEST.COM for HTTP/127.6.209.212@KRBTEST.COM
Sep 26 15:17:47 dist-test-slave-j6q7 krb5kdc[18832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:17:47.737963 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44211
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:44211
--ranger_config_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.209.212:54783/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:17:48.029986 19276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:17:48.030236 19276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:17:48.030405 19276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:17:48.036695 19276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:17:48.036804 19276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:17:48.036895 19276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:17:48.036988 19276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:17:48.037068 19276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:17:48.045611 19276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.209.212:54783/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:44211
--ranger_config_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:17:48.046586 19276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:17:48.047662 19276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:17:48.057909 19282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:48.057940 19284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:48.058104 19281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:17:48.058568 19276 server_base.cc:1034] running on GCE node
I20260926 15:17:48.059123 19276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:17:48.060283 19276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:17:48.061461 19276 hybrid_clock.cc:648] HybridClock initialized: now 1790435868061428 us; error 31 us; skew 500 ppm
I20260926 15:17:48.061725 19276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:17:48 dist-test-slave-j6q7 krb5kdc[18832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435868, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:17:48.064365 19276 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:17:48.065609 19276 webserver.cc:460] Webserver started at http://127.6.209.254:46821/ using document root <none> and password file <none>
I20260926 15:17:48.066195 19276 fs_manager.cc:359] Metadata directory not provided
I20260926 15:17:48.066308 19276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:17:48.066576 19276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 15:17:48 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435868, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for HTTP/127.6.209.212@KRBTEST.COM
I20260926 15:17:48.122474 19276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "eea5cedf631c46e4ae7dda3dd4e6e3a4"
format_stamp: "Formatted at 2026-09-26 15:17:48 on dist-test-slave-j6q7"
server_key: "8338672188e462e1db0085d75e122584"
server_key_iv: "f0f11ee7056b6ef0de0b3d3025316e77"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:48.123239 19276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "eea5cedf631c46e4ae7dda3dd4e6e3a4"
format_stamp: "Formatted at 2026-09-26 15:17:48 on dist-test-slave-j6q7"
server_key: "8338672188e462e1db0085d75e122584"
server_key_iv: "f0f11ee7056b6ef0de0b3d3025316e77"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:48.128208 19276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 15:17:48.146201 19291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:48.147588 19276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:17:48.147755 19276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "eea5cedf631c46e4ae7dda3dd4e6e3a4"
format_stamp: "Formatted at 2026-09-26 15:17:48 on dist-test-slave-j6q7"
server_key: "8338672188e462e1db0085d75e122584"
server_key_iv: "f0f11ee7056b6ef0de0b3d3025316e77"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:48.147956 19276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:17:48.172328 19276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:17:48.175571 19276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:17:48.175863 19276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:17:48.192718 19276 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:44211
I20260926 15:17:48.192737 19342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:44211 every 8 connection(s)
I20260926 15:17:48.193892 19276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:17:48.197160 19343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:17:48.197728 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 19276
I20260926 15:17:48.197933 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:17:48.198791 6983 external_mini_cluster.cc:1428] Setting key a9124d0ba2ce48cbf12aaffd74380fae
I20260926 15:17:48.210273 19343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4: Bootstrap starting.
Sep 26 15:17:48 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:17:48.214097 19343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4: Neither blocks nor log segments found. Creating new log.
I20260926 15:17:48.215057 19343 log.cc:824] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4: Log is configured to *not* fsync() on all Append() calls
I20260926 15:17:48.217660 19343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4: No bootstrap required, opened a new log
I20260926 15:17:48.220463 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:48.200704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40904 (local address 127.6.209.254:44211)
0926 15:17:48.201512 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:48.201524 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:48.201561 (+ 37us) server_negotiation.cc:407] Connection header received
0926 15:17:48.203132 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:48.203151 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:48.203493 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:48.203780 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:48.205206 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:48.206360 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:48.208173 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:48.208750 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:48.208929 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:48.213537 (+ 4608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:48.213545 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:48.213557 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:48.213588 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:48.216349 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:48.217070 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:48.217078 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:48.217090 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:48.217249 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:48.217674 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:48.217678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:48.217680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:48.218157 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:48.218555 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:48.219022 (+ 467us) server_negotiation.cc:299] Negotiation successful
0926 15:17:48.219425 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":370,"threads_started":1}
I20260926 15:17:48.225636 19343 raft_consensus.cc:348] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea5cedf631c46e4ae7dda3dd4e6e3a4" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44211 } }
I20260926 15:17:48.225967 19343 raft_consensus.cc:374] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:17:48.226068 19343 raft_consensus.cc:729] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea5cedf631c46e4ae7dda3dd4e6e3a4, State: Initialized, Role: FOLLOWER
I20260926 15:17:48.226604 19343 consensus_queue.cc:260] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea5cedf631c46e4ae7dda3dd4e6e3a4" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44211 } }
I20260926 15:17:48.226832 19343 raft_consensus.cc:388] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:17:48.226966 19343 raft_consensus.cc:482] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:17:48.227119 19343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:17:48.229450 19343 raft_consensus.cc:504] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea5cedf631c46e4ae7dda3dd4e6e3a4" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44211 } }
I20260926 15:17:48.229944 19343 leader_election.cc:302] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eea5cedf631c46e4ae7dda3dd4e6e3a4; no voters:
I20260926 15:17:48.230609 19343 leader_election.cc:288] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:17:48.230643 19348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:17:48.232257 19348 raft_consensus.cc:686] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [term 1 LEADER]: Becoming Leader. State: Replica: eea5cedf631c46e4ae7dda3dd4e6e3a4, State: Running, Role: LEADER
I20260926 15:17:48.232730 19348 consensus_queue.cc:237] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea5cedf631c46e4ae7dda3dd4e6e3a4" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44211 } }
I20260926 15:17:48.232982 19343 sys_catalog.cc:564] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:17:48.235222 19350 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eea5cedf631c46e4ae7dda3dd4e6e3a4. Latest consensus state: current_term: 1 leader_uuid: "eea5cedf631c46e4ae7dda3dd4e6e3a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea5cedf631c46e4ae7dda3dd4e6e3a4" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44211 } } }
I20260926 15:17:48.235244 19349 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eea5cedf631c46e4ae7dda3dd4e6e3a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea5cedf631c46e4ae7dda3dd4e6e3a4" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44211 } } }
I20260926 15:17:48.235648 19349 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [sys.catalog]: This master's current role is: LEADER
I20260926 15:17:48.235648 19350 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [sys.catalog]: This master's current role is: LEADER
I20260926 15:17:48.236493 19353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:17:48.240754 19353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:17:48.248411 19353 catalog_manager.cc:1347] Generated new cluster ID: 757e2f5eed8748008126646ba9f5d984
I20260926 15:17:48.248553 19353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:17:48.248730 19343 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 15:17:48.275205 19353 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:17:48.277066 19353 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:17:48.287237 19353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4: Generated new TSK 0
I20260926 15:17:48.288164 19353 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
I20260926 15:17:48.948796 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.209.212:54783/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44211
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:17:49 dist-test-slave-j6q7 krb5kdc[18832](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790435869, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 15:17:49.333899 19388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:17:49.334283 19388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:17:49.334478 19388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:17:49.345405 19388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:17:49.345527 19388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:17:49.345681 19388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:17:49.360271 19388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.209.212:54783/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:17:49.361867 19388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:17:49.363399 19388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:17:49.380578 19398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:49.393206 19399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:17:49.403970 19388 server_base.cc:1034] running on GCE node
W20260926 15:17:49.404204 19401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:17:49.405073 19388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:17:49.412451 19388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:17:49.415760 19388 hybrid_clock.cc:648] HybridClock initialized: now 1790435869415694 us; error 54 us; skew 500 ppm
I20260926 15:17:49.416100 19388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:17:49 dist-test-slave-j6q7 krb5kdc[18832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435869, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:17:49.419020 19388 init.cc:377] Logged in from keytab as kudu/127.6.209.193@KRBTEST.COM (short username kudu)
I20260926 15:17:49.420608 19388 webserver.cc:460] Webserver started at http://127.6.209.193:38183/ using document root <none> and password file <none>
I20260926 15:17:49.421373 19388 fs_manager.cc:359] Metadata directory not provided
I20260926 15:17:49.421500 19388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:17:49.421808 19388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 15:17:49 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435869, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for HTTP/127.6.209.212@KRBTEST.COM
I20260926 15:17:49.441061 19388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "d2270f0acf6a4d54b7342dac337b1de9"
format_stamp: "Formatted at 2026-09-26 15:17:49 on dist-test-slave-j6q7"
server_key: "2a03cbf2dda19eb1a975043aca66e192"
server_key_iv: "e9951ee07c44b547019bea06131daf35"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:49.442101 19388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "d2270f0acf6a4d54b7342dac337b1de9"
format_stamp: "Formatted at 2026-09-26 15:17:49 on dist-test-slave-j6q7"
server_key: "2a03cbf2dda19eb1a975043aca66e192"
server_key_iv: "e9951ee07c44b547019bea06131daf35"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:49.458451 19388 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.015s sys 0.000s
I20260926 15:17:49.485455 19408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:49.487278 19388 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.010s
I20260926 15:17:49.487498 19388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "d2270f0acf6a4d54b7342dac337b1de9"
format_stamp: "Formatted at 2026-09-26 15:17:49 on dist-test-slave-j6q7"
server_key: "2a03cbf2dda19eb1a975043aca66e192"
server_key_iv: "e9951ee07c44b547019bea06131daf35"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:49.487794 19388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:17:49.540124 19388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:17:49.544865 19388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:17:49.545217 19388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:17:49.552417 19388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:17:49.560302 19388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:17:49.560438 19388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:49.560618 19388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:17:49.560732 19388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 15:17:49 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435869, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for HTTP/127.6.209.212@KRBTEST.COM
I20260926 15:17:50.140113 19388 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:34317
I20260926 15:17:50.140188 19521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:34317 every 8 connection(s)
I20260926 15:17:50.141782 19388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:17:50.144101 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 19388
I20260926 15:17:50.144300 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:17:50.145170 6983 external_mini_cluster.cc:1428] Setting key 0029e1d8f78bb49b835f2e10e04ccbb8
Sep 26 15:17:50 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435869, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for HTTP/127.6.209.212@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.194@KRBTEST.COM" created.
Sep 26 15:17:50 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435869, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:17:50.198056 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:50.157091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44211 (local address 127.6.209.193:60293)
0926 15:17:50.169360 (+ 12269us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:50.169422 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:50.171188 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:50.172154 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:50.172166 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:50.172714 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:17:50.173477 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:50.173501 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:50.175371 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:50.175377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:50.177169 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:50.177183 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:50.180052 (+ 2869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:50.180064 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:50.180408 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:50.181196 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:17:50.181230 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:17:50.183908 (+ 2678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:17:50.192521 (+ 8613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:50.192531 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:50.192548 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:50.193426 (+ 878us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:50.194032 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:50.194037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:50.194040 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:50.194387 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:50.195146 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:17:50.195153 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:17:50.195609 (+ 456us) client_negotiation.cc:769] Sending connection context
0926 15:17:50.195912 (+ 303us) client_negotiation.cc:240] Negotiation successful
0926 15:17:50.196609 (+ 697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11992,"thread_start_us":11783,"threads_started":1}
I20260926 15:17:50.199842 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:50.157564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:60293 (local address 127.6.209.254:44211)
0926 15:17:50.168785 (+ 11221us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:50.168793 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:50.171449 (+ 2656us) server_negotiation.cc:407] Connection header received
0926 15:17:50.171545 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:50.171550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:50.171637 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:50.171728 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:50.173649 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:50.175103 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:50.177325 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:50.179293 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:50.179401 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:50.184129 (+ 4728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:50.184136 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:50.184143 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:50.184183 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:50.192234 (+ 8051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:50.193629 (+ 1395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:50.193636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:50.193642 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:50.193856 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:50.194584 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:50.194587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:50.194590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:50.195017 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:50.199116 (+ 4099us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:50.199251 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 15:17:50.199468 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11059,"thread_start_us":10723,"threads_started":1}
I20260926 15:17:50.201663 19522 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44211
I20260926 15:17:50.201956 19522 heartbeater.cc:464] Registering TS with master...
I20260926 15:17:50.202692 19522 heartbeater.cc:511] Master 127.6.209.254:44211 requested a full tablet report, sending...
I20260926 15:17:50.205147 19308 ts_manager.cc:194] Registered new tserver with Master: d2270f0acf6a4d54b7342dac337b1de9 (127.6.209.193:34317)
I20260926 15:17:50.206487 19308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.193@KRBTEST.COM'} at 127.6.209.193:60293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
I20260926 15:17:50.213693 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.209.212:54783/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44211
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:17:50.555964 19529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:17:50.556197 19529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:17:50.556278 19529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:17:50.562630 19529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:17:50.562750 19529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:17:50.562922 19529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:17:50.571708 19529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.209.212:54783/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:17:50.572708 19529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:17:50.573801 19529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:17:50.585981 19544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:50.586017 19542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:50.586104 19541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:17:50.586604 19529 server_base.cc:1034] running on GCE node
I20260926 15:17:50.587085 19529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:17:50.588001 19529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:17:50.589182 19529 hybrid_clock.cc:648] HybridClock initialized: now 1790435870589149 us; error 31 us; skew 500 ppm
I20260926 15:17:50.589416 19529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:17:50 dist-test-slave-j6q7 krb5kdc[18832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435870, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:17:50.592226 19529 init.cc:377] Logged in from keytab as kudu/127.6.209.194@KRBTEST.COM (short username kudu)
I20260926 15:17:50.593458 19529 webserver.cc:460] Webserver started at http://127.6.209.194:39979/ using document root <none> and password file <none>
I20260926 15:17:50.594060 19529 fs_manager.cc:359] Metadata directory not provided
I20260926 15:17:50.594149 19529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:17:50.594363 19529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 15:17:50 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435870, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for HTTP/127.6.209.212@KRBTEST.COM
I20260926 15:17:50.609406 19529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "84ed829b45a84752a17f0e5733205d71"
format_stamp: "Formatted at 2026-09-26 15:17:50 on dist-test-slave-j6q7"
server_key: "728108d3b9fa1fcdfb887e482684ae48"
server_key_iv: "2314dbc77f9d3bb2d046e594f79a6dc5"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:50.610390 19529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "84ed829b45a84752a17f0e5733205d71"
format_stamp: "Formatted at 2026-09-26 15:17:50 on dist-test-slave-j6q7"
server_key: "728108d3b9fa1fcdfb887e482684ae48"
server_key_iv: "2314dbc77f9d3bb2d046e594f79a6dc5"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:50.616315 19529 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260926 15:17:50.629771 19551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:50.631147 19529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 15:17:50.631304 19529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "84ed829b45a84752a17f0e5733205d71"
format_stamp: "Formatted at 2026-09-26 15:17:50 on dist-test-slave-j6q7"
server_key: "728108d3b9fa1fcdfb887e482684ae48"
server_key_iv: "2314dbc77f9d3bb2d046e594f79a6dc5"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:50.631518 19529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:17:50.655640 19529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:17:50.658864 19529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:17:50.659116 19529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:17:50.659916 19529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:17:50.661249 19529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:17:50.661329 19529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:50.661486 19529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:17:50.661533 19529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:50.692973 19529 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:36123
I20260926 15:17:50.693066 19663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:36123 every 8 connection(s)
I20260926 15:17:50.694103 19529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:17:50.699398 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 19529
I20260926 15:17:50.699559 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:17:50.700130 6983 external_mini_cluster.cc:1428] Setting key 58ab22f993d035e7d1a254620cae8462
Sep 26 15:17:50 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435870, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:17:50.713572 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:50.696345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:55607 (local address 127.6.209.254:44211)
0926 15:17:50.696502 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:50.696507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:50.697606 (+ 1099us) server_negotiation.cc:407] Connection header received
0926 15:17:50.699097 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:50.699101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:50.699170 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:50.699251 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:50.701060 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:50.701808 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:50.703865 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:50.704193 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:50.704279 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:50.707889 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:50.707893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:50.707895 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:50.707923 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:50.710895 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:50.711576 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:50.711584 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:50.711589 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:50.711719 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:50.712103 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:50.712111 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:50.712113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:50.712323 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:50.712477 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:50.713080 (+ 603us) server_negotiation.cc:299] Negotiation successful
0926 15:17:50.713276 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 15:17:50.714350 19666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:50.696785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44211 (local address 127.6.209.194:55607)
0926 15:17:50.697456 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:50.697487 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:50.698873 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:50.699504 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:50.699515 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:50.700074 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:17:50.700864 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:50.700894 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:50.701967 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:50.701973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:50.703736 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:50.703752 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:50.704393 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:50.704401 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:50.704660 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:50.705270 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:17:50.705303 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:17:50.707703 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:17:50.711040 (+ 3337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:50.711046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:50.711072 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:50.711406 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:50.711822 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:50.711825 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:50.711827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:50.711978 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:50.712441 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:17:50.712447 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:17:50.712724 (+ 277us) client_negotiation.cc:769] Sending connection context
0926 15:17:50.713000 (+ 276us) client_negotiation.cc:240] Negotiation successful
0926 15:17:50.713405 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":255,"threads_started":1}
I20260926 15:17:50.715515 19664 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44211
I20260926 15:17:50.715770 19664 heartbeater.cc:464] Registering TS with master...
I20260926 15:17:50.716424 19664 heartbeater.cc:511] Master 127.6.209.254:44211 requested a full tablet report, sending...
I20260926 15:17:50.717784 19308 ts_manager.cc:194] Registered new tserver with Master: 84ed829b45a84752a17f0e5733205d71 (127.6.209.194:36123)
I20260926 15:17:50.718930 19308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.194@KRBTEST.COM'} at 127.6.209.194:55607
WARNING: no policy specified for kudu/127.6.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
I20260926 15:17:50.747591 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.209.212:54783/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44211
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:17:51.040432 19669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:17:51.040663 19669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:17:51.040812 19669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:17:51.047566 19669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:17:51.047668 19669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:17:51.047820 19669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:17:51.055956 19669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.209.212:54783/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:17:51.057091 19669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:17:51.058214 19669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:17:51.069903 19675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:51.070114 19674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:51.070442 19677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:17:51.070822 19669 server_base.cc:1034] running on GCE node
I20260926 15:17:51.071367 19669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:17:51.072144 19669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:17:51.073354 19669 hybrid_clock.cc:648] HybridClock initialized: now 1790435871073280 us; error 66 us; skew 500 ppm
I20260926 15:17:51.073613 19669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:17:51 dist-test-slave-j6q7 krb5kdc[18832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435871, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:17:51.076171 19669 init.cc:377] Logged in from keytab as kudu/127.6.209.195@KRBTEST.COM (short username kudu)
I20260926 15:17:51.077337 19669 webserver.cc:460] Webserver started at http://127.6.209.195:43741/ using document root <none> and password file <none>
I20260926 15:17:51.077879 19669 fs_manager.cc:359] Metadata directory not provided
I20260926 15:17:51.078011 19669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:17:51.078289 19669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 15:17:51 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435871, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for HTTP/127.6.209.212@KRBTEST.COM
I20260926 15:17:51.090559 19669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "140574e11b654ce7bc27e1c2a6e49138"
format_stamp: "Formatted at 2026-09-26 15:17:51 on dist-test-slave-j6q7"
server_key: "023332b19991422f28fcc3ed635dd41c"
server_key_iv: "02fa0e6b351816bea5a226869ebc2b48"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:51.091526 19669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "140574e11b654ce7bc27e1c2a6e49138"
format_stamp: "Formatted at 2026-09-26 15:17:51 on dist-test-slave-j6q7"
server_key: "023332b19991422f28fcc3ed635dd41c"
server_key_iv: "02fa0e6b351816bea5a226869ebc2b48"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:51.097314 19669 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 15:17:51.111629 19684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:51.112990 19669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:17:51.113178 19669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "140574e11b654ce7bc27e1c2a6e49138"
format_stamp: "Formatted at 2026-09-26 15:17:51 on dist-test-slave-j6q7"
server_key: "023332b19991422f28fcc3ed635dd41c"
server_key_iv: "02fa0e6b351816bea5a226869ebc2b48"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:51.113334 19669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:17:51.127444 19669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:17:51.131313 19669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:17:51.131623 19669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:17:51.132463 19669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:17:51.133814 19669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:17:51.133903 19669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:51.134009 19669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:17:51.134089 19669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:51.162757 19669 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:44399
I20260926 15:17:51.162789 19796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:44399 every 8 connection(s)
I20260926 15:17:51.163848 19669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:17:51.165002 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 19669
I20260926 15:17:51.165159 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:17:51.165582 6983 external_mini_cluster.cc:1428] Setting key 2819189bb3bb680502d6e9c74977fe36
Sep 26 15:17:51 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435871, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:17:51.181970 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:51.166515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:51503 (local address 127.6.209.254:44211)
0926 15:17:51.166706 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:51.166712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:51.167850 (+ 1138us) server_negotiation.cc:407] Connection header received
0926 15:17:51.169611 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:51.169614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:51.169685 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:51.169759 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:51.171107 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:51.171758 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:51.173143 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:51.173527 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:51.173618 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:51.176337 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:51.176341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:51.176344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:51.176371 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:51.178777 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:51.179609 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:51.179615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:51.179619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:51.179740 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:51.180260 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:51.180266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:51.180269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:51.180657 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:51.180798 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:51.181444 (+ 646us) server_negotiation.cc:299] Negotiation successful
0926 15:17:51.181657 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 15:17:51.182717 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:51.166790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44211 (local address 127.6.209.195:51503)
0926 15:17:51.167678 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:51.167727 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:51.169386 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:51.169997 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:51.170005 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:51.170386 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:17:51.170892 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:51.170908 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:51.171881 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:51.171884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:51.173040 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:51.173049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:51.173694 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:51.173697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:51.173852 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:51.174377 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:17:51.174394 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:17:51.176162 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:17:51.178917 (+ 2755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:51.178925 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:51.178938 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:51.179521 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:51.179870 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:51.179878 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:51.179884 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:51.180146 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:51.180775 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:17:51.180782 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:17:51.181086 (+ 304us) client_negotiation.cc:769] Sending connection context
0926 15:17:51.181354 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 15:17:51.181787 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":466,"threads_started":1}
I20260926 15:17:51.183818 19797 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44211
I20260926 15:17:51.184087 19797 heartbeater.cc:464] Registering TS with master...
I20260926 15:17:51.184681 19797 heartbeater.cc:511] Master 127.6.209.254:44211 requested a full tablet report, sending...
I20260926 15:17:51.185914 19308 ts_manager.cc:194] Registered new tserver with Master: 140574e11b654ce7bc27e1c2a6e49138 (127.6.209.195:44399)
I20260926 15:17:51.186666 19308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.195@KRBTEST.COM'} at 127.6.209.195:51503
I20260926 15:17:51.192433 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:17:51.194084 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 19388
Sep 26 15:17:51 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.193@KRBTEST.COM
I20260926 15:17:51.209146 19801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:51.195018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57110 (local address 127.6.209.193:34317)
0926 15:17:51.195669 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:51.195675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:51.195695 (+ 20us) server_negotiation.cc:407] Connection header received
0926 15:17:51.195745 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:51.195750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:51.195898 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:51.196005 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:51.197649 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:51.198535 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:51.200456 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:51.201027 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:51.201118 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:51.204785 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:51.204794 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:51.204803 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:51.204891 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:51.207087 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:51.207568 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:51.207573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:51.207575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:51.207680 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:51.207939 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:51.207942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:51.207944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:51.208179 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:51.208375 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:51.208597 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 15:17:51.208747 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":364,"threads_started":1}
I20260926 15:17:51.209695 19522 heartbeater.cc:503] Master 127.6.209.254:44211 was elected leader, sending a full tablet report...
I20260926 15:17:51.210420 19516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:17:51.286485 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 19388
I20260926 15:17:51.300654 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 19529
Sep 26 15:17:51 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.194@KRBTEST.COM
I20260926 15:17:51.319718 19803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:51.301514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42668 (local address 127.6.209.194:36123)
0926 15:17:51.302043 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:51.302049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:51.302071 (+ 22us) server_negotiation.cc:407] Connection header received
0926 15:17:51.302137 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:51.302143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:51.302333 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:51.302444 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:51.303454 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:51.304324 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:51.305683 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:51.306151 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:51.306240 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:51.315185 (+ 8945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:51.315194 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:51.315206 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:51.315233 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:51.317575 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:51.318218 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:51.318224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:51.318226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:51.318334 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:51.318606 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:51.318609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:51.318610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:51.318873 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:51.319062 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:51.319273 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 15:17:51.319453 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":310,"threads_started":1}
I20260926 15:17:51.321002 19659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:17:51.399493 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 19529
I20260926 15:17:51.414278 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 19669
Sep 26 15:17:51 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.195@KRBTEST.COM
I20260926 15:17:51.429181 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:51.415173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54534 (local address 127.6.209.195:44399)
0926 15:17:51.415798 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:51.415808 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:51.415843 (+ 35us) server_negotiation.cc:407] Connection header received
0926 15:17:51.415896 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:51.415902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:51.416110 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:51.416269 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:51.417523 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:51.418411 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:51.419613 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:51.420180 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:51.420322 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:51.424335 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:51.424344 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:51.424356 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:51.424386 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:51.426693 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:51.427302 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:51.427309 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:51.427311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:51.427421 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:51.427776 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:51.427782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:51.427785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:51.428079 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:51.428376 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:51.428632 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 15:17:51.428891 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":342,"threads_started":1}
I20260926 15:17:51.430346 19792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:17:51.506505 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 19669
I20260926 15:17:51.520802 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 19276
I20260926 15:17:51.522107 19338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:17:51.615684 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 19276
I20260926 15:17:51.637323 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44211
--webserver_interface=127.6.209.254
--webserver_port=46821
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:44211
--ranger_config_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.209.212:54783/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.209.212:54783/kms with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:17:51.926343 19808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:17:51.926661 19808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:17:51.926849 19808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:17:51.933912 19808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:17:51.934087 19808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:17:51.934202 19808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:17:51.934296 19808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:17:51.934387 19808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:17:51.943163 19808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.209.212:54783/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:44211
--ranger_config_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=46821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:17:51.944170 19808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:17:51.945309 19808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:17:51.955615 19813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:51.955590 19814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:51.955845 19816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:17:51.956413 19808 server_base.cc:1034] running on GCE node
I20260926 15:17:51.957111 19808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:17:51.958315 19808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:17:51.959501 19808 hybrid_clock.cc:648] HybridClock initialized: now 1790435871959472 us; error 25 us; skew 500 ppm
I20260926 15:17:51.959811 19808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:17:51 dist-test-slave-j6q7 krb5kdc[18832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435871, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:17:51.962632 19808 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:17:51.963876 19808 webserver.cc:460] Webserver started at http://127.6.209.254:46821/ using document root <none> and password file <none>
I20260926 15:17:51.964430 19808 fs_manager.cc:359] Metadata directory not provided
I20260926 15:17:51.964584 19808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 15:17:51 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435871, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for HTTP/127.6.209.212@KRBTEST.COM
I20260926 15:17:52.016407 19808 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:17:52.018982 19824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:52.020212 19808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 15:17:52.020397 19808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "eea5cedf631c46e4ae7dda3dd4e6e3a4"
format_stamp: "Formatted at 2026-09-26 15:17:48 on dist-test-slave-j6q7"
server_key: "8338672188e462e1db0085d75e122584"
server_key_iv: "f0f11ee7056b6ef0de0b3d3025316e77"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:52.021391 19808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:17:52.043682 19808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:17:52.046694 19808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:17:52.046967 19808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:17:52.063138 19808 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:44211
I20260926 15:17:52.063145 19875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:44211 every 8 connection(s)
I20260926 15:17:52.064308 19808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:17:52.066177 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 19808
I20260926 15:17:52.067121 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.209.212:54783/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:34317
--local_ip_for_outbound_sockets=127.6.209.193
--tserver_master_addrs=127.6.209.254:44211
--webserver_port=38183
--webserver_interface=127.6.209.193
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.209.212:54783/kms with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:17:52.072499 19876 sys_catalog.cc:263] Verifying existing consensus state
I20260926 15:17:52.075910 19876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4: Bootstrap starting.
I20260926 15:17:52.092345 19876 log.cc:824] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4: Log is configured to *not* fsync() on all Append() calls
I20260926 15:17:52.099887 19876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:17:52.100419 19876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4: Bootstrap complete.
I20260926 15:17:52.108547 19876 raft_consensus.cc:348] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea5cedf631c46e4ae7dda3dd4e6e3a4" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44211 } }
I20260926 15:17:52.109321 19876 raft_consensus.cc:729] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea5cedf631c46e4ae7dda3dd4e6e3a4, State: Initialized, Role: FOLLOWER
I20260926 15:17:52.109877 19876 consensus_queue.cc:260] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea5cedf631c46e4ae7dda3dd4e6e3a4" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44211 } }
I20260926 15:17:52.110113 19876 raft_consensus.cc:388] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:17:52.110245 19876 raft_consensus.cc:482] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:17:52.110437 19876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [term 1 FOLLOWER]: Advancing to term 2
I20260926 15:17:52.113884 19876 raft_consensus.cc:504] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea5cedf631c46e4ae7dda3dd4e6e3a4" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44211 } }
I20260926 15:17:52.114348 19876 leader_election.cc:302] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eea5cedf631c46e4ae7dda3dd4e6e3a4; no voters:
I20260926 15:17:52.114956 19876 leader_election.cc:288] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 15:17:52.115085 19880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 15:17:52.116272 19880 raft_consensus.cc:686] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [term 2 LEADER]: Becoming Leader. State: Replica: eea5cedf631c46e4ae7dda3dd4e6e3a4, State: Running, Role: LEADER
I20260926 15:17:52.116861 19880 consensus_queue.cc:237] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea5cedf631c46e4ae7dda3dd4e6e3a4" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44211 } }
I20260926 15:17:52.117035 19876 sys_catalog.cc:564] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:17:52.119212 19882 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eea5cedf631c46e4ae7dda3dd4e6e3a4. Latest consensus state: current_term: 2 leader_uuid: "eea5cedf631c46e4ae7dda3dd4e6e3a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea5cedf631c46e4ae7dda3dd4e6e3a4" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44211 } } }
I20260926 15:17:52.119540 19882 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [sys.catalog]: This master's current role is: LEADER
I20260926 15:17:52.119441 19881 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "eea5cedf631c46e4ae7dda3dd4e6e3a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea5cedf631c46e4ae7dda3dd4e6e3a4" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44211 } } }
I20260926 15:17:52.119819 19881 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4 [sys.catalog]: This master's current role is: LEADER
I20260926 15:17:52.129757 19887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:17:52.135649 19887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:17:52.137301 19887 catalog_manager.cc:1259] Loaded cluster ID: 757e2f5eed8748008126646ba9f5d984
I20260926 15:17:52.137454 19887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:17:52.142155 19887 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:17:52.143801 19887 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4: Loaded TSK: 0
I20260926 15:17:52.145531 19887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:17:52.147974 19876 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260926 15:17:52.476125 19878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:17:52.476537 19878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:17:52.476739 19878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:17:52.487337 19878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:17:52.487448 19878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:17:52.487650 19878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:17:52.502377 19878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.209.212:54783/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:34317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=38183
--tserver_master_addrs=127.6.209.254:44211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:17:52.503798 19878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:17:52.505358 19878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:17:52.541833 19914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:52.549450 19917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:52.549674 19913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:17:52.555207 19878 server_base.cc:1034] running on GCE node
I20260926 15:17:52.556085 19878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:17:52.561061 19878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:17:52.568511 19878 hybrid_clock.cc:648] HybridClock initialized: now 1790435872568416 us; error 91 us; skew 500 ppm
I20260926 15:17:52.568878 19878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:17:52 dist-test-slave-j6q7 krb5kdc[18832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435872, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:17:52.572196 19878 init.cc:377] Logged in from keytab as kudu/127.6.209.193@KRBTEST.COM (short username kudu)
I20260926 15:17:52.575227 19878 webserver.cc:460] Webserver started at http://127.6.209.193:38183/ using document root <none> and password file <none>
I20260926 15:17:52.575875 19878 fs_manager.cc:359] Metadata directory not provided
I20260926 15:17:52.576012 19878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 15:17:52 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435872, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for HTTP/127.6.209.212@KRBTEST.COM
I20260926 15:17:52.610340 19878 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.007s sys 0.007s
I20260926 15:17:52.618024 19925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:52.619725 19878 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.004s
I20260926 15:17:52.619947 19878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "d2270f0acf6a4d54b7342dac337b1de9"
format_stamp: "Formatted at 2026-09-26 15:17:49 on dist-test-slave-j6q7"
server_key: "2a03cbf2dda19eb1a975043aca66e192"
server_key_iv: "e9951ee07c44b547019bea06131daf35"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:52.621519 19878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:17:52.650441 19878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:17:52.656677 19878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:17:52.657039 19878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:17:52.658159 19878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:17:52.660158 19878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:17:52.660315 19878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:52.660492 19878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:17:52.660638 19878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:52.736901 19878 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:34317
I20260926 15:17:52.736955 20037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:34317 every 8 connection(s)
I20260926 15:17:52.738417 19878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:17:52.745606 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 19878
I20260926 15:17:52.746340 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.209.212:54783/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:36123
--local_ip_for_outbound_sockets=127.6.209.194
--tserver_master_addrs=127.6.209.254:44211
--webserver_port=39979
--webserver_interface=127.6.209.194
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.209.212:54783/kms with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:17:52 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435872, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:17:52.764335 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:52.742488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44211 (local address 127.6.209.193:46985)
0926 15:17:52.743229 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:52.743268 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:52.745044 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:52.746658 (+ 1614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:52.746668 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:52.747244 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:17:52.747715 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:52.747732 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:52.749690 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:52.749695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:52.750913 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:52.750924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:52.751906 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:52.751912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:52.752144 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:52.752770 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:17:52.752789 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:17:52.755526 (+ 2737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:17:52.759758 (+ 4232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:52.759765 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:52.759778 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:52.760355 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:52.761224 (+ 869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:52.761230 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:52.761246 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:52.761495 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:52.762155 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:17:52.762163 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:17:52.762450 (+ 287us) client_negotiation.cc:769] Sending connection context
0926 15:17:52.762708 (+ 258us) client_negotiation.cc:240] Negotiation successful
0926 15:17:52.763169 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":326,"threads_started":1}
I20260926 15:17:52.777200 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:52.742567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:46985 (local address 127.6.209.254:44211)
0926 15:17:52.743956 (+ 1389us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:52.743971 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:52.744019 (+ 48us) server_negotiation.cc:407] Connection header received
0926 15:17:52.745781 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:52.745800 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:52.746176 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:52.746518 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:52.747896 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:52.749553 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:52.751059 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:52.751687 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:52.751891 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:52.755793 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:52.755804 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:52.755819 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:52.755876 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:52.759610 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:52.760505 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:52.760513 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:52.760520 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:52.761108 (+ 588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:52.761607 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:52.761612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:52.761615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:52.762016 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:52.762212 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:52.775655 (+ 13443us) server_negotiation.cc:299] Negotiation successful
0926 15:17:52.776036 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1146,"thread_start_us":951,"threads_started":1}
I20260926 15:17:52.780761 20038 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44211
I20260926 15:17:52.781097 20038 heartbeater.cc:464] Registering TS with master...
I20260926 15:17:52.781860 20038 heartbeater.cc:511] Master 127.6.209.254:44211 requested a full tablet report, sending...
W20260926 15:17:52.784584 20038 heartbeater.cc:644] Failed to heartbeat to 127.6.209.254:44211 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 15:17:52.784767 20038 heartbeater.cc:464] Registering TS with master...
I20260926 15:17:52.785006 20038 heartbeater.cc:511] Master 127.6.209.254:44211 requested a full tablet report, sending...
I20260926 15:17:52.786042 20038 heartbeater.cc:464] Registering TS with master...
I20260926 15:17:52.786248 20038 heartbeater.cc:511] Master 127.6.209.254:44211 requested a full tablet report, sending...
W20260926 15:17:52.787160 20038 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 15:17:53.120491 20042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:17:53.120800 20042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:17:53.121004 20042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:17:53.127949 20042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:17:53.128074 20042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:17:53.128242 20042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:17:53.137252 20042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.209.212:54783/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:36123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=39979
--tserver_master_addrs=127.6.209.254:44211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:17:53.138682 20042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:17:53.140393 20042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:17:53.177176 20056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:53.179937 20057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:53.181937 20061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:17:53.184401 20042 server_base.cc:1034] running on GCE node
I20260926 15:17:53.185122 20042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:17:53.186293 20042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:17:53.187606 20042 hybrid_clock.cc:648] HybridClock initialized: now 1790435873187547 us; error 45 us; skew 500 ppm
I20260926 15:17:53.188051 20042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:17:53 dist-test-slave-j6q7 krb5kdc[18832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435873, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:17:53.192365 20042 init.cc:377] Logged in from keytab as kudu/127.6.209.194@KRBTEST.COM (short username kudu)
I20260926 15:17:53.214957 20042 webserver.cc:460] Webserver started at http://127.6.209.194:39979/ using document root <none> and password file <none>
I20260926 15:17:53.215677 20042 fs_manager.cc:359] Metadata directory not provided
I20260926 15:17:53.215830 20042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 15:17:53 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435873, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for HTTP/127.6.209.212@KRBTEST.COM
I20260926 15:17:53.243248 20042 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 15:17:53.246292 20069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:53.247736 20042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 15:17:53.247937 20042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "84ed829b45a84752a17f0e5733205d71"
format_stamp: "Formatted at 2026-09-26 15:17:50 on dist-test-slave-j6q7"
server_key: "728108d3b9fa1fcdfb887e482684ae48"
server_key_iv: "2314dbc77f9d3bb2d046e594f79a6dc5"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:53.249066 20042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 26 15:17:53 dist-test-slave-j6q7 krb5kdc[18832](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790435873, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:17:53.273294 20042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:17:53.278038 20042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:17:53.278395 20042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:17:53.292207 20042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:17:53.293812 20042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:17:53.293969 20042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:53.294137 20042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:17:53.294261 20042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:53.331753 20042 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:36123
I20260926 15:17:53.331779 20181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:36123 every 8 connection(s)
I20260926 15:17:53.333365 20042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:17:53.334424 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 20042
I20260926 15:17:53.335165 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.209.212:54783/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:44399
--local_ip_for_outbound_sockets=127.6.209.195
--tserver_master_addrs=127.6.209.254:44211
--webserver_port=43741
--webserver_interface=127.6.209.195
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.209.212:54783/kms with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:17:53 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435873, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:17:53.365025 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:53.340971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:35281 (local address 127.6.209.254:44211)
0926 15:17:53.343227 (+ 2256us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:53.343235 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:53.344120 (+ 885us) server_negotiation.cc:407] Connection header received
0926 15:17:53.346632 (+ 2512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:53.346638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:53.346715 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:53.346791 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:53.348782 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:53.349803 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:53.352215 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:53.352847 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:53.352982 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:53.357347 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:53.357354 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:53.357359 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:53.357408 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:53.361231 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:53.362241 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:53.362247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:53.362251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:53.362428 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:53.362997 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:53.363002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:53.363006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:53.363358 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:53.363492 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:53.364403 (+ 911us) server_negotiation.cc:299] Negotiation successful
0926 15:17:53.364609 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2088,"thread_start_us":1999,"threads_started":1}
I20260926 15:17:53.366206 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:53.336527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44211 (local address 127.6.209.194:35281)
0926 15:17:53.343733 (+ 7206us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:53.343782 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:53.346335 (+ 2553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:53.347239 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:53.347250 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:53.347792 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:17:53.348561 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:53.348598 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:53.350001 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:53.350006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:53.352044 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:53.352058 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:53.353244 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:53.353251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:53.353532 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:53.354311 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:17:53.354339 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:17:53.357088 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:17:53.361435 (+ 4347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:53.361446 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:53.361465 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:53.362061 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:53.362572 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:53.362577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:53.362582 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:53.362848 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:53.363523 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:17:53.363533 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:17:53.363945 (+ 412us) client_negotiation.cc:769] Sending connection context
0926 15:17:53.364258 (+ 313us) client_negotiation.cc:240] Negotiation successful
0926 15:17:53.364831 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6967,"thread_start_us":6776,"threads_started":1}
I20260926 15:17:53.367671 20182 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44211
I20260926 15:17:53.367976 20182 heartbeater.cc:464] Registering TS with master...
I20260926 15:17:53.368705 20182 heartbeater.cc:511] Master 127.6.209.254:44211 requested a full tablet report, sending...
I20260926 15:17:53.370637 19841 ts_manager.cc:194] Registered new tserver with Master: 84ed829b45a84752a17f0e5733205d71 (127.6.209.194:36123)
I20260926 15:17:53.372715 19841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.194@KRBTEST.COM'} at 127.6.209.194:35281
Sep 26 15:17:53 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435873, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for HTTP/127.6.209.212@KRBTEST.COM
W20260926 15:17:53.732880 20186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:17:53.733189 20186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:17:53.733362 20186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:17:53.742839 20186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:17:53.742973 20186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:17:53.743145 20186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:17:53.751909 20186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.209.212:54783/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:44399
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=43741
--tserver_master_addrs=127.6.209.254:44211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:17:53.753144 20186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:17:53.754611 20186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:17:53.772724 20192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:53.777411 20193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:53.785995 20195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:17:53.786675 20186 server_base.cc:1034] running on GCE node
I20260926 15:17:53.787269 20186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:17:53.787325 20038 heartbeater.cc:464] Registering TS with master...
I20260926 15:17:53.787694 20038 heartbeater.cc:511] Master 127.6.209.254:44211 requested a full tablet report, sending...
I20260926 15:17:53.788735 19841 ts_manager.cc:194] Registered new tserver with Master: d2270f0acf6a4d54b7342dac337b1de9 (127.6.209.193:34317)
I20260926 15:17:53.789721 19841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.193@KRBTEST.COM'} at 127.6.209.193:46985
I20260926 15:17:53.793071 20186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:17:53.794381 20186 hybrid_clock.cc:648] HybridClock initialized: now 1790435873794311 us; error 46 us; skew 500 ppm
I20260926 15:17:53.794729 20186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:17:53 dist-test-slave-j6q7 krb5kdc[18832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435873, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:17:53.798947 20186 init.cc:377] Logged in from keytab as kudu/127.6.209.195@KRBTEST.COM (short username kudu)
I20260926 15:17:53.803134 20186 webserver.cc:460] Webserver started at http://127.6.209.195:43741/ using document root <none> and password file <none>
I20260926 15:17:53.803836 20186 fs_manager.cc:359] Metadata directory not provided
I20260926 15:17:53.803994 20186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 15:17:53 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435873, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for HTTP/127.6.209.212@KRBTEST.COM
I20260926 15:17:53.823210 20186 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 15:17:53.826078 20203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:53.827507 20186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:17:53.827677 20186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "140574e11b654ce7bc27e1c2a6e49138"
format_stamp: "Formatted at 2026-09-26 15:17:51 on dist-test-slave-j6q7"
server_key: "023332b19991422f28fcc3ed635dd41c"
server_key_iv: "02fa0e6b351816bea5a226869ebc2b48"
server_key_version: "kuduclusterkey@0"
I20260926 15:17:53.828572 20186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:17:53.862404 20186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:17:53.866083 20186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:17:53.866328 20186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:17:53.867301 20186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:17:53.870556 20186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:17:53.870687 20186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:53.870820 20186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:17:53.870918 20186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:53.911993 20186 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:44399
I20260926 15:17:53.912066 20315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:44399 every 8 connection(s)
I20260926 15:17:53.913709 20186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:17:53.921023 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 20186
Sep 26 15:17:53 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435873, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:17:53.947034 20318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:53.916558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44211 (local address 127.6.209.195:33325)
0926 15:17:53.920120 (+ 3562us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:53.920173 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:53.922817 (+ 2644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:53.925903 (+ 3086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:53.925912 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:53.926415 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:17:53.927152 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:53.927240 (+ 88us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:53.929029 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:53.929033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:53.930882 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:53.930895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:53.935226 (+ 4331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:53.935234 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:53.935482 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:53.936215 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:17:53.936241 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:17:53.938536 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:17:53.942710 (+ 4174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:53.942718 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:53.942732 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:53.943252 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:53.943655 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:53.943660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:53.943663 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:53.943878 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:53.944421 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:17:53.944428 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:17:53.944787 (+ 359us) client_negotiation.cc:769] Sending connection context
0926 15:17:53.945451 (+ 664us) client_negotiation.cc:240] Negotiation successful
0926 15:17:53.945902 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3268,"thread_start_us":3071,"threads_started":1}
I20260926 15:17:53.947705 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:53.917049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:33325 (local address 127.6.209.254:44211)
0926 15:17:53.925214 (+ 8165us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:53.925222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:53.925243 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:17:53.925305 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:53.925309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:53.925391 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:53.925495 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:53.927476 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:53.928873 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:53.932006 (+ 3133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:53.932930 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:53.933022 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:53.938785 (+ 5763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:53.938792 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:53.938796 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:53.938863 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:53.942549 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:53.943380 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:53.943385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:53.943389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:53.943536 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:53.943998 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:53.944002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:53.944004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:53.944279 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:53.944908 (+ 629us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:53.945298 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 15:17:53.947389 (+ 2091us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8030,"thread_start_us":7905,"threads_started":1}
I20260926 15:17:53.949384 20316 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44211
I20260926 15:17:53.949667 20316 heartbeater.cc:464] Registering TS with master...
I20260926 15:17:53.950089 20321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:53.922085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47608 (local address 127.6.209.254:44211)
0926 15:17:53.931640 (+ 9555us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:53.931644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:53.931845 (+ 201us) server_negotiation.cc:407] Connection header received
0926 15:17:53.931955 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:53.931959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:53.932042 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:53.932138 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:53.935371 (+ 3233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:53.936756 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:53.938610 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:53.939330 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:53.939456 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:53.943420 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:53.943427 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:53.943431 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:53.943466 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:53.946593 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:53.947380 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:53.947385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:53.947389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:53.947555 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:53.947953 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:53.947957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:53.947959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:53.948246 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:53.948507 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:53.949572 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0926 15:17:53.949755 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9457,"thread_start_us":9341,"threads_started":1}
I20260926 15:17:53.950459 20316 heartbeater.cc:511] Master 127.6.209.254:44211 requested a full tablet report, sending...
I20260926 15:17:53.952350 19840 ts_manager.cc:194] Registered new tserver with Master: 140574e11b654ce7bc27e1c2a6e49138 (127.6.209.195:44399)
I20260926 15:17:53.953572 19840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.195@KRBTEST.COM'} at 127.6.209.195:33325
I20260926 15:17:53.964800 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:17:53.985617 20321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:53.970321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47622 (local address 127.6.209.254:44211)
0926 15:17:53.970490 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:53.970496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:53.970969 (+ 473us) server_negotiation.cc:407] Connection header received
0926 15:17:53.971212 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:53.971218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:53.971288 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:53.971352 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:53.973319 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:53.974235 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:53.975910 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:53.976452 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:53.976564 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:53.979411 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:53.979416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:53.979420 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:53.979453 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:53.982571 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:53.983362 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:53.983368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:53.983372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:53.983520 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:53.983960 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:53.983964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:53.983967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:53.984272 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:53.984497 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:53.985052 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 15:17:53.985252 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 15:17:53 dist-test-slave-j6q7 krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435873, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for HTTP/127.6.209.212@KRBTEST.COM
I20260926 15:17:53.991072 19840 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:17:54.001230 19840 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:17:54.103507 20334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:54.025664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:36123 (local address 127.0.0.1:32986)
0926 15:17:54.077392 (+ 51728us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:54.077443 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:54.077669 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:54.091205 (+ 13536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:54.091210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:54.091237 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:17:54.091527 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:54.091536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.097166 (+ 5630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.097170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:54.100583 (+ 3413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:54.100595 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.102648 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.102655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:54.102841 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:54.102896 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 15:17:54.103005 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 15:17:54.103176 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51483,"thread_start_us":53850,"threads_started":1}
I20260926 15:17:54.104393 20333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:54.025192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32986 (local address 127.6.209.194:36123)
0926 15:17:54.076888 (+ 51696us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:54.076897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:54.088185 (+ 11288us) server_negotiation.cc:407] Connection header received
0926 15:17:54.088284 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:54.088290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:54.088518 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:54.088682 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:17:54.092621 (+ 3939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.094089 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.100741 (+ 6652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.102164 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.102302 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:54.102488 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:54.103902 (+ 1414us) server_negotiation.cc:299] Negotiation successful
0926 15:17:54.104092 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51493,"thread_start_us":54984,"threads_started":1}
I20260926 15:17:54.105918 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:54.081074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37704 (local address 127.6.209.193:34317)
0926 15:17:54.095539 (+ 14465us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:54.095546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:54.095569 (+ 23us) server_negotiation.cc:407] Connection header received
0926 15:17:54.095651 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:54.095657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:54.095850 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:54.096007 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:17:54.096827 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.098242 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.102864 (+ 4622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.104195 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.104284 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:54.104469 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:54.105168 (+ 699us) server_negotiation.cc:299] Negotiation successful
0926 15:17:54.105450 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14310,"thread_start_us":15767,"threads_started":1}
I20260926 15:17:54.106637 20338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:54.087657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:34317 (local address 127.0.0.1:37704)
0926 15:17:54.094975 (+ 7318us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:54.094992 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:54.095092 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:54.096225 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:54.096229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:54.096265 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:17:54.096627 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:54.096636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.098406 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.098411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:54.102700 (+ 4289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:54.102711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.104629 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.104637 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:54.104800 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:54.104866 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 15:17:54.106249 (+ 1383us) client_negotiation.cc:240] Negotiation successful
0926 15:17:54.106359 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7213,"thread_start_us":9348,"threads_started":1}
I20260926 15:17:54.108575 20117 tablet_service.cc:1467] Processing CreateTablet for tablet d657fbfd6a03411ab1fcf4858681b404 (DEFAULT_TABLE table=test-table [id=c320455041c443cc894a8cd3e8e6557d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:17:54.109948 20117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d657fbfd6a03411ab1fcf4858681b404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:17:54.110056 19973 tablet_service.cc:1467] Processing CreateTablet for tablet d657fbfd6a03411ab1fcf4858681b404 (DEFAULT_TABLE table=test-table [id=c320455041c443cc894a8cd3e8e6557d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:17:54.111030 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:54.034915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58620 (local address 127.6.209.195:44399)
0926 15:17:54.078722 (+ 43807us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:54.078730 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:54.078761 (+ 31us) server_negotiation.cc:407] Connection header received
0926 15:17:54.078851 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:54.078868 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:54.079128 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:54.079298 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:17:54.092948 (+ 13650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.094497 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.097554 (+ 3057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.098893 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.099029 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:54.099238 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:54.110531 (+ 11293us) server_negotiation.cc:299] Negotiation successful
0926 15:17:54.110695 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43650,"thread_start_us":51873,"threads_started":1}
I20260926 15:17:54.111397 19973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d657fbfd6a03411ab1fcf4858681b404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:17:54.112851 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:54.034418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:44399 (local address 127.0.0.1:58620)
0926 15:17:54.078180 (+ 43762us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:54.078194 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:54.078284 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:54.089004 (+ 10720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:54.089010 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:54.089050 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:17:54.089382 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:54.089393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.094596 (+ 5203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.094602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:54.097404 (+ 2802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:54.097415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.110160 (+ 12745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.110167 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:54.110331 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:54.110362 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 15:17:54.111252 (+ 890us) client_negotiation.cc:240] Negotiation successful
0926 15:17:54.111389 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43656,"thread_start_us":45942,"threads_started":1}
I20260926 15:17:54.115869 20251 tablet_service.cc:1467] Processing CreateTablet for tablet d657fbfd6a03411ab1fcf4858681b404 (DEFAULT_TABLE table=test-table [id=c320455041c443cc894a8cd3e8e6557d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:17:54.118147 20251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d657fbfd6a03411ab1fcf4858681b404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:17:54.137475 20339 tablet_bootstrap.cc:492] T d657fbfd6a03411ab1fcf4858681b404 P 84ed829b45a84752a17f0e5733205d71: Bootstrap starting.
I20260926 15:17:54.139249 20341 tablet_bootstrap.cc:492] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138: Bootstrap starting.
I20260926 15:17:54.139849 20340 tablet_bootstrap.cc:492] T d657fbfd6a03411ab1fcf4858681b404 P d2270f0acf6a4d54b7342dac337b1de9: Bootstrap starting.
I20260926 15:17:54.142287 20341 tablet_bootstrap.cc:654] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138: Neither blocks nor log segments found. Creating new log.
I20260926 15:17:54.143141 20340 tablet_bootstrap.cc:654] T d657fbfd6a03411ab1fcf4858681b404 P d2270f0acf6a4d54b7342dac337b1de9: Neither blocks nor log segments found. Creating new log.
I20260926 15:17:54.143412 20341 log.cc:824] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138: Log is configured to *not* fsync() on all Append() calls
I20260926 15:17:54.144958 20339 tablet_bootstrap.cc:654] T d657fbfd6a03411ab1fcf4858681b404 P 84ed829b45a84752a17f0e5733205d71: Neither blocks nor log segments found. Creating new log.
I20260926 15:17:54.146018 20341 tablet_bootstrap.cc:492] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138: No bootstrap required, opened a new log
I20260926 15:17:54.146240 20341 ts_tablet_manager.cc:1397] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 15:17:54.150314 20339 log.cc:824] T d657fbfd6a03411ab1fcf4858681b404 P 84ed829b45a84752a17f0e5733205d71: Log is configured to *not* fsync() on all Append() calls
I20260926 15:17:54.153786 20340 log.cc:824] T d657fbfd6a03411ab1fcf4858681b404 P d2270f0acf6a4d54b7342dac337b1de9: Log is configured to *not* fsync() on all Append() calls
I20260926 15:17:54.155159 20341 raft_consensus.cc:348] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ed829b45a84752a17f0e5733205d71" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36123 } } peers { permanent_uuid: "140574e11b654ce7bc27e1c2a6e49138" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44399 } } peers { permanent_uuid: "d2270f0acf6a4d54b7342dac337b1de9" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 34317 } }
I20260926 15:17:54.155771 20341 raft_consensus.cc:374] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:17:54.155915 20341 raft_consensus.cc:729] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 140574e11b654ce7bc27e1c2a6e49138, State: Initialized, Role: FOLLOWER
I20260926 15:17:54.156625 20341 consensus_queue.cc:260] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ed829b45a84752a17f0e5733205d71" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36123 } } peers { permanent_uuid: "140574e11b654ce7bc27e1c2a6e49138" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44399 } } peers { permanent_uuid: "d2270f0acf6a4d54b7342dac337b1de9" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 34317 } }
I20260926 15:17:54.173089 20340 tablet_bootstrap.cc:492] T d657fbfd6a03411ab1fcf4858681b404 P d2270f0acf6a4d54b7342dac337b1de9: No bootstrap required, opened a new log
I20260926 15:17:54.175014 20340 ts_tablet_manager.cc:1397] T d657fbfd6a03411ab1fcf4858681b404 P d2270f0acf6a4d54b7342dac337b1de9: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.000s
I20260926 15:17:54.176680 20339 tablet_bootstrap.cc:492] T d657fbfd6a03411ab1fcf4858681b404 P 84ed829b45a84752a17f0e5733205d71: No bootstrap required, opened a new log
I20260926 15:17:54.176964 20339 ts_tablet_manager.cc:1397] T d657fbfd6a03411ab1fcf4858681b404 P 84ed829b45a84752a17f0e5733205d71: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.004s
I20260926 15:17:54.201552 20316 heartbeater.cc:503] Master 127.6.209.254:44211 was elected leader, sending a full tablet report...
I20260926 15:17:54.202287 20341 ts_tablet_manager.cc:1428] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138: Time spent starting tablet: real 0.056s user 0.033s sys 0.023s
I20260926 15:17:54.209915 20340 raft_consensus.cc:348] T d657fbfd6a03411ab1fcf4858681b404 P d2270f0acf6a4d54b7342dac337b1de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ed829b45a84752a17f0e5733205d71" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36123 } } peers { permanent_uuid: "140574e11b654ce7bc27e1c2a6e49138" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44399 } } peers { permanent_uuid: "d2270f0acf6a4d54b7342dac337b1de9" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 34317 } }
I20260926 15:17:54.210368 20340 raft_consensus.cc:374] T d657fbfd6a03411ab1fcf4858681b404 P d2270f0acf6a4d54b7342dac337b1de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:17:54.210475 20340 raft_consensus.cc:729] T d657fbfd6a03411ab1fcf4858681b404 P d2270f0acf6a4d54b7342dac337b1de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2270f0acf6a4d54b7342dac337b1de9, State: Initialized, Role: FOLLOWER
I20260926 15:17:54.211112 20340 consensus_queue.cc:260] T d657fbfd6a03411ab1fcf4858681b404 P d2270f0acf6a4d54b7342dac337b1de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ed829b45a84752a17f0e5733205d71" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36123 } } peers { permanent_uuid: "140574e11b654ce7bc27e1c2a6e49138" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44399 } } peers { permanent_uuid: "d2270f0acf6a4d54b7342dac337b1de9" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 34317 } }
I20260926 15:17:54.200510 20339 raft_consensus.cc:348] T d657fbfd6a03411ab1fcf4858681b404 P 84ed829b45a84752a17f0e5733205d71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ed829b45a84752a17f0e5733205d71" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36123 } } peers { permanent_uuid: "140574e11b654ce7bc27e1c2a6e49138" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44399 } } peers { permanent_uuid: "d2270f0acf6a4d54b7342dac337b1de9" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 34317 } }
I20260926 15:17:54.211778 20339 raft_consensus.cc:374] T d657fbfd6a03411ab1fcf4858681b404 P 84ed829b45a84752a17f0e5733205d71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:17:54.211871 20339 raft_consensus.cc:729] T d657fbfd6a03411ab1fcf4858681b404 P 84ed829b45a84752a17f0e5733205d71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84ed829b45a84752a17f0e5733205d71, State: Initialized, Role: FOLLOWER
I20260926 15:17:54.212388 20339 consensus_queue.cc:260] T d657fbfd6a03411ab1fcf4858681b404 P 84ed829b45a84752a17f0e5733205d71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ed829b45a84752a17f0e5733205d71" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36123 } } peers { permanent_uuid: "140574e11b654ce7bc27e1c2a6e49138" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44399 } } peers { permanent_uuid: "d2270f0acf6a4d54b7342dac337b1de9" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 34317 } }
I20260926 15:17:54.212648 20038 heartbeater.cc:503] Master 127.6.209.254:44211 was elected leader, sending a full tablet report...
I20260926 15:17:54.213860 20340 ts_tablet_manager.cc:1428] T d657fbfd6a03411ab1fcf4858681b404 P d2270f0acf6a4d54b7342dac337b1de9: Time spent starting tablet: real 0.031s user 0.002s sys 0.009s
I20260926 15:17:54.217619 20182 heartbeater.cc:503] Master 127.6.209.254:44211 was elected leader, sending a full tablet report...
I20260926 15:17:54.218080 20339 ts_tablet_manager.cc:1428] T d657fbfd6a03411ab1fcf4858681b404 P 84ed829b45a84752a17f0e5733205d71: Time spent starting tablet: real 0.041s user 0.007s sys 0.007s
W20260926 15:17:54.242728 20039 tablet.cc:2369] T d657fbfd6a03411ab1fcf4858681b404 P d2270f0acf6a4d54b7342dac337b1de9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:17:54.352211 20183 tablet.cc:2369] T d657fbfd6a03411ab1fcf4858681b404 P 84ed829b45a84752a17f0e5733205d71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:17:54.352397 20345 raft_consensus.cc:482] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:17:54.352684 20345 raft_consensus.cc:504] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ed829b45a84752a17f0e5733205d71" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36123 } } peers { permanent_uuid: "140574e11b654ce7bc27e1c2a6e49138" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44399 } } peers { permanent_uuid: "d2270f0acf6a4d54b7342dac337b1de9" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 34317 } }
I20260926 15:17:54.354172 20345 leader_election.cc:288] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84ed829b45a84752a17f0e5733205d71 (127.6.209.194:36123), d2270f0acf6a4d54b7342dac337b1de9 (127.6.209.193:34317)
I20260926 15:17:54.360513 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:54.354591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:34363 (local address 127.6.209.193:34317)
0926 15:17:54.354772 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:54.354777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:54.354814 (+ 37us) server_negotiation.cc:407] Connection header received
0926 15:17:54.355021 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:54.355025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:54.355093 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:54.355185 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:17:54.355586 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.356452 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.358649 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.359708 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.359858 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:54.359924 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:54.360182 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 15:17:54.360295 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:17:54.360585 20318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:54.354562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:34317 (local address 127.6.209.195:34363)
0926 15:17:54.354735 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:54.354754 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:54.354869 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:54.355230 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:54.355236 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:54.355272 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:17:54.355472 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:54.355480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.356586 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.356589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:54.358515 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:54.358528 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.359842 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.359845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:54.359957 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:54.360003 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 15:17:54.360140 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 15:17:54.360292 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":97}
I20260926 15:17:54.360886 20333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:54.354517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:40285 (local address 127.6.209.194:36123)
0926 15:17:54.354687 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:54.354692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:54.355329 (+ 637us) server_negotiation.cc:407] Connection header received
0926 15:17:54.355443 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:54.355445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:54.355490 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:54.355585 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:17:54.356005 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.356763 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.358839 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.359924 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.360043 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:54.360200 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:54.360476 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 15:17:54.360622 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 15:17:54.361209 20355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:54.354557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:36123 (local address 127.6.209.195:40285)
0926 15:17:54.355230 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:54.355244 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:54.355344 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:54.355619 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:54.355622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:54.355639 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:17:54.355868 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:54.355874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.356910 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.356915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:54.358712 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:54.358720 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.360143 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.360150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:54.360321 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:54.360349 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 15:17:54.360434 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 15:17:54.360541 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":166,"thread_start_us":243,"threads_started":1}
I20260926 15:17:54.361598 19993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d657fbfd6a03411ab1fcf4858681b404" candidate_uuid: "140574e11b654ce7bc27e1c2a6e49138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2270f0acf6a4d54b7342dac337b1de9" is_pre_election: true
I20260926 15:17:54.362144 19993 raft_consensus.cc:2454] T d657fbfd6a03411ab1fcf4858681b404 P d2270f0acf6a4d54b7342dac337b1de9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 140574e11b654ce7bc27e1c2a6e49138 in term 0.
I20260926 15:17:54.362149 20137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d657fbfd6a03411ab1fcf4858681b404" candidate_uuid: "140574e11b654ce7bc27e1c2a6e49138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84ed829b45a84752a17f0e5733205d71" is_pre_election: true
I20260926 15:17:54.362646 20137 raft_consensus.cc:2454] T d657fbfd6a03411ab1fcf4858681b404 P 84ed829b45a84752a17f0e5733205d71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 140574e11b654ce7bc27e1c2a6e49138 in term 0.
I20260926 15:17:54.362864 20206 leader_election.cc:302] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 140574e11b654ce7bc27e1c2a6e49138, d2270f0acf6a4d54b7342dac337b1de9; no voters:
I20260926 15:17:54.363245 20345 raft_consensus.cc:2781] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:17:54.363377 20345 raft_consensus.cc:482] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:17:54.363509 20345 raft_consensus.cc:3037] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:17:54.365406 20345 raft_consensus.cc:504] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ed829b45a84752a17f0e5733205d71" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36123 } } peers { permanent_uuid: "140574e11b654ce7bc27e1c2a6e49138" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44399 } } peers { permanent_uuid: "d2270f0acf6a4d54b7342dac337b1de9" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 34317 } }
I20260926 15:17:54.365998 20345 leader_election.cc:288] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138 [CANDIDATE]: Term 1 election: Requested vote from peers 84ed829b45a84752a17f0e5733205d71 (127.6.209.194:36123), d2270f0acf6a4d54b7342dac337b1de9 (127.6.209.193:34317)
I20260926 15:17:54.366277 20137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d657fbfd6a03411ab1fcf4858681b404" candidate_uuid: "140574e11b654ce7bc27e1c2a6e49138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84ed829b45a84752a17f0e5733205d71"
I20260926 15:17:54.366393 19993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d657fbfd6a03411ab1fcf4858681b404" candidate_uuid: "140574e11b654ce7bc27e1c2a6e49138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2270f0acf6a4d54b7342dac337b1de9"
I20260926 15:17:54.366459 20137 raft_consensus.cc:3037] T d657fbfd6a03411ab1fcf4858681b404 P 84ed829b45a84752a17f0e5733205d71 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:17:54.366586 19993 raft_consensus.cc:3037] T d657fbfd6a03411ab1fcf4858681b404 P d2270f0acf6a4d54b7342dac337b1de9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:17:54.369326 20137 raft_consensus.cc:2454] T d657fbfd6a03411ab1fcf4858681b404 P 84ed829b45a84752a17f0e5733205d71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 140574e11b654ce7bc27e1c2a6e49138 in term 1.
I20260926 15:17:54.369326 19993 raft_consensus.cc:2454] T d657fbfd6a03411ab1fcf4858681b404 P d2270f0acf6a4d54b7342dac337b1de9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 140574e11b654ce7bc27e1c2a6e49138 in term 1.
I20260926 15:17:54.369650 20205 leader_election.cc:302] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 140574e11b654ce7bc27e1c2a6e49138, 84ed829b45a84752a17f0e5733205d71; no voters:
I20260926 15:17:54.369992 20345 raft_consensus.cc:2781] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:17:54.370509 20345 raft_consensus.cc:686] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138 [term 1 LEADER]: Becoming Leader. State: Replica: 140574e11b654ce7bc27e1c2a6e49138, State: Running, Role: LEADER
I20260926 15:17:54.370934 20345 consensus_queue.cc:237] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ed829b45a84752a17f0e5733205d71" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36123 } } peers { permanent_uuid: "140574e11b654ce7bc27e1c2a6e49138" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44399 } } peers { permanent_uuid: "d2270f0acf6a4d54b7342dac337b1de9" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 34317 } }
I20260926 15:17:54.375018 19840 catalog_manager.cc:5495] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138 reported cstate change: term changed from 0 to 1, leader changed from <none> to 140574e11b654ce7bc27e1c2a6e49138 (127.6.209.195). New cstate: current_term: 1 leader_uuid: "140574e11b654ce7bc27e1c2a6e49138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84ed829b45a84752a17f0e5733205d71" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "140574e11b654ce7bc27e1c2a6e49138" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2270f0acf6a4d54b7342dac337b1de9" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 34317 } health_report { overall_health: UNKNOWN } } }
I20260926 15:17:54.414945 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:54.410261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58622 (local address 127.6.209.195:44399)
0926 15:17:54.410483 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:54.410488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:54.410506 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:17:54.410647 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:54.410650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:54.410720 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:54.410834 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:17:54.411151 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.411829 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.413185 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.413567 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.413647 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:54.414017 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:17:54.414450 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:17:54.414542 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:54.414633 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 15:17:54.414686 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 15:17:54.422875 19993 raft_consensus.cc:1261] T d657fbfd6a03411ab1fcf4858681b404 P d2270f0acf6a4d54b7342dac337b1de9 [term 1 FOLLOWER]: Refusing update from remote peer 140574e11b654ce7bc27e1c2a6e49138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:17:54.422902 20137 raft_consensus.cc:1261] T d657fbfd6a03411ab1fcf4858681b404 P 84ed829b45a84752a17f0e5733205d71 [term 1 FOLLOWER]: Refusing update from remote peer 140574e11b654ce7bc27e1c2a6e49138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:17:54.423786 20356 consensus_queue.cc:1035] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2270f0acf6a4d54b7342dac337b1de9" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 34317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:17:54.424132 20345 consensus_queue.cc:1035] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84ed829b45a84752a17f0e5733205d71" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 15:17:54.428247 20317 tablet.cc:2369] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:17:54.434755 20359 mvcc.cc:205] Tried to move back new op lower bound from 7333625341625896960 to 7333625341426872320. Current Snapshot: MvccSnapshot[applied={T|T < 7333625341625896960}]
I20260926 15:17:54.447523 19840 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:17:54.447773 19840 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:17:54.451128 19840 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eea5cedf631c46e4ae7dda3dd4e6e3a4: Sending DeleteTablet for 3 replicas of tablet d657fbfd6a03411ab1fcf4858681b404
I20260926 15:17:54.452111 20117 tablet_service.cc:1514] Processing DeleteTablet for tablet d657fbfd6a03411ab1fcf4858681b404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:17:54 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:32986
I20260926 15:17:54.453088 20369 tablet_replica.cc:330] T d657fbfd6a03411ab1fcf4858681b404 P 84ed829b45a84752a17f0e5733205d71: stopping tablet replica
I20260926 15:17:54.453018 20251 tablet_service.cc:1514] Processing DeleteTablet for tablet d657fbfd6a03411ab1fcf4858681b404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:17:54 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:58620
I20260926 15:17:54.453457 20369 raft_consensus.cc:2229] T d657fbfd6a03411ab1fcf4858681b404 P 84ed829b45a84752a17f0e5733205d71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:17:54.453928 20369 raft_consensus.cc:2258] T d657fbfd6a03411ab1fcf4858681b404 P 84ed829b45a84752a17f0e5733205d71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:17:54.454664 20370 tablet_replica.cc:330] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138: stopping tablet replica
I20260926 15:17:54.454831 19973 tablet_service.cc:1514] Processing DeleteTablet for tablet d657fbfd6a03411ab1fcf4858681b404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:17:54 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:37704
I20260926 15:17:54.455014 20370 raft_consensus.cc:2229] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:17:54.455377 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 19878
I20260926 15:17:54.455482 20370 raft_consensus.cc:2258] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:17:54.460506 20369 ts_tablet_manager.cc:1905] T d657fbfd6a03411ab1fcf4858681b404 P 84ed829b45a84752a17f0e5733205d71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:17:54.466987 20369 ts_tablet_manager.cc:1918] T d657fbfd6a03411ab1fcf4858681b404 P 84ed829b45a84752a17f0e5733205d71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:17:54.467159 20369 log.cc:1196] T d657fbfd6a03411ab1fcf4858681b404 P 84ed829b45a84752a17f0e5733205d71: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/d657fbfd6a03411ab1fcf4858681b404
I20260926 15:17:54.467653 20369 ts_tablet_manager.cc:1939] T d657fbfd6a03411ab1fcf4858681b404 P 84ed829b45a84752a17f0e5733205d71: Deleting consensus metadata
I20260926 15:17:54.469882 19826 catalog_manager.cc:4841] TS 84ed829b45a84752a17f0e5733205d71 (127.6.209.194:36123): tablet d657fbfd6a03411ab1fcf4858681b404 (table test-table [id=c320455041c443cc894a8cd3e8e6557d]) successfully deleted
I20260926 15:17:54.470292 20370 ts_tablet_manager.cc:1905] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:17:54.471081 20371 tablet_replica.cc:330] T d657fbfd6a03411ab1fcf4858681b404 P d2270f0acf6a4d54b7342dac337b1de9: stopping tablet replica
I20260926 15:17:54.472615 20371 raft_consensus.cc:2229] T d657fbfd6a03411ab1fcf4858681b404 P d2270f0acf6a4d54b7342dac337b1de9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:17:54.473448 20371 raft_consensus.cc:2258] T d657fbfd6a03411ab1fcf4858681b404 P d2270f0acf6a4d54b7342dac337b1de9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:17:54.474893 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:54.456065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37716 (local address 127.6.209.193:34317)
0926 15:17:54.456369 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:54.456375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:54.456769 (+ 394us) server_negotiation.cc:407] Connection header received
0926 15:17:54.456869 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:54.456873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:54.456937 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:54.457028 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:54.461462 (+ 4434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.462163 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.464495 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.465024 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.465156 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:54.468900 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:54.468911 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:54.468924 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:54.468966 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:54.472215 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:54.472989 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:54.472996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:54.472999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:54.473121 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:54.473535 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:54.473540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:54.473543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:54.473875 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:54.474149 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:54.474408 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 15:17:54.474593 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260926 15:17:54.475596 20033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:17:54.476557 20370 ts_tablet_manager.cc:1918] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:17:54.476689 20370 log.cc:1196] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/d657fbfd6a03411ab1fcf4858681b404
I20260926 15:17:54.477136 20370 ts_tablet_manager.cc:1939] T d657fbfd6a03411ab1fcf4858681b404 P 140574e11b654ce7bc27e1c2a6e49138: Deleting consensus metadata
I20260926 15:17:54.478690 19827 catalog_manager.cc:4841] TS 140574e11b654ce7bc27e1c2a6e49138 (127.6.209.195:44399): tablet d657fbfd6a03411ab1fcf4858681b404 (table test-table [id=c320455041c443cc894a8cd3e8e6557d]) successfully deleted
I20260926 15:17:54.556910 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 19878
I20260926 15:17:54.572654 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 20042
W20260926 15:17:54.572654 19827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.209.193:34317 (error 108)
I20260926 15:17:54.575407 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:54.574907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:34317 (local address 127.0.0.1:37722)
0926 15:17:54.575091 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:54.575229 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:34317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 15:17:54.575744 19827 catalog_manager.cc:4584] TS d2270f0acf6a4d54b7342dac337b1de9 (127.6.209.193:34317): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d657fbfd6a03411ab1fcf4858681b404: Network error: Client connection negotiation failed: client connection to 127.6.209.193:34317: connect: Connection refused (error 111)
I20260926 15:17:54.587431 20333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:54.573505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32996 (local address 127.6.209.194:36123)
0926 15:17:54.573691 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:54.573697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:54.573718 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:17:54.573879 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:54.573885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:54.573965 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:54.574078 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:54.575616 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.576624 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.577874 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.578521 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.578689 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:54.581143 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:54.581161 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:54.581180 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:54.581230 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:54.584587 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:54.585409 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:54.585420 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:54.585425 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:54.585588 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:54.585918 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:54.585923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:54.585926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:54.586391 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:54.586664 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:54.586858 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 15:17:54.587090 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 15:17:54.588097 20177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:17:54.629408 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:54.628922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:34317 (local address 127.0.0.1:37736)
0926 15:17:54.629119 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:54.629231 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:34317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 15:17:54.630620 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:54.630147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:34317 (local address 127.0.0.1:37752)
0926 15:17:54.630377 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:54.630462 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:34317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260926 15:17:54.675524 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 20042
I20260926 15:17:54.691303 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:54.690707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:34317 (local address 127.0.0.1:37756)
0926 15:17:54.690969 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:54.691095 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:34317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 15:17:54.691697 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 20186
I20260926 15:17:54.693168 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:54.692283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:34317 (local address 127.0.0.1:37768)
0926 15:17:54.692954 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:54.693016 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:34317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573}
I20260926 15:17:54.707181 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:54.692391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58638 (local address 127.6.209.195:44399)
0926 15:17:54.692604 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:54.692611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:54.692683 (+ 72us) server_negotiation.cc:407] Connection header received
0926 15:17:54.692892 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:54.692899 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:54.692983 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:54.693109 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:54.694312 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.695495 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.696787 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:54.697609 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:54.697749 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:54.700472 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:54.700487 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:54.700504 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:54.700541 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:54.704402 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:54.705119 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:54.705127 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:54.705130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:54.705300 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:54.705655 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:54.705659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:54.705662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:54.706048 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:54.706377 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:54.706616 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 15:17:54.706805 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:17:54.707923 20311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:17:54.773667 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:54.773136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:34317 (local address 127.0.0.1:37778)
0926 15:17:54.773361 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:54.773473 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:34317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 15:17:54.774767 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:54.774331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:34317 (local address 127.0.0.1:37792)
0926 15:17:54.774543 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:54.774632 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:34317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 15:17:54.816623 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 20186
I20260926 15:17:54.835263 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 19808
I20260926 15:17:54.836270 19871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:17:54.937539 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 19808
I20260926 15:17:54.959107 6983 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 15:17:55.294680 6983 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 15:17:55.294873 6983 mini_ranger.cc:66] Stopping Ranger...
I20260926 15:17:55.640717 6983 mini_ranger.cc:68] Stopped Ranger
2026-09-26 15:17:55.641 UTC [18862] LOG: received smart shutdown request
2026-09-26 15:17:55.644 UTC [18862] LOG: background worker "logical replication launcher" (PID 18871) exited with exit code 1
2026-09-26 15:17:55.645 UTC [18866] LOG: shutting down
2026-09-26 15:17:55.798 UTC [18862] LOG: database system is shut down
2026-09-26T15:17:55Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46600 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:17:56 dist-test-slave-j6q7 krb5kdc[20386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:17:56 dist-test-slave-j6q7 krb5kdc[20386](info): set up 2 sockets
Sep 26 15:17:56 dist-test-slave-j6q7 krb5kdc[20386](info): commencing operation
krb5kdc: starting...
W20260926 15:17:56.916654 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:17:57 dist-test-slave-j6q7 krb5kdc[20386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:17:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:17:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:17:57.102491 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:37383
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:36477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:37383,127.6.209.253:42813,127.6.209.252:38169
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:17:57.386489 20399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:17:57.386759 20399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:17:57.386911 20399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:17:57.392910 20399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:17:57.393025 20399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:17:57.393121 20399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:17:57.393213 20399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:17:57.393271 20399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:17:57.402701 20399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:36477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:37383,127.6.209.253:42813,127.6.209.252:38169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:37383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:17:57.403743 20399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:17:57.404917 20399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:17:57.415555 20405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:57.415606 20404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:57.415642 20407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:17:57.416190 20399 server_base.cc:1034] running on GCE node
I20260926 15:17:57.416882 20399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:17:57.418061 20399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:17:57.419272 20399 hybrid_clock.cc:648] HybridClock initialized: now 1790435877419230 us; error 40 us; skew 500 ppm
I20260926 15:17:57.419566 20399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:17:57 dist-test-slave-j6q7 krb5kdc[20386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435877, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:17:57.422528 20399 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:17:57.423751 20399 webserver.cc:460] Webserver started at http://127.6.209.254:32817/ using document root <none> and password file <none>
I20260926 15:17:57.424340 20399 fs_manager.cc:359] Metadata directory not provided
I20260926 15:17:57.424458 20399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:17:57.424728 20399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:17:57.427191 20399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "3005514e2a9e44a8ac3bfe4815a909ab"
format_stamp: "Formatted at 2026-09-26 15:17:57 on dist-test-slave-j6q7"
server_key: "d85e366abf37a37ba467afa17c6167d6"
server_key_iv: "abbb2c74a5bf4d1afb130acac1591695"
server_key_version: "encryptionkey@0"
I20260926 15:17:57.427863 20399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "3005514e2a9e44a8ac3bfe4815a909ab"
format_stamp: "Formatted at 2026-09-26 15:17:57 on dist-test-slave-j6q7"
server_key: "d85e366abf37a37ba467afa17c6167d6"
server_key_iv: "abbb2c74a5bf4d1afb130acac1591695"
server_key_version: "encryptionkey@0"
I20260926 15:17:57.432399 20399 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 15:17:57.435243 20414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:57.436225 20399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:17:57.436367 20399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "3005514e2a9e44a8ac3bfe4815a909ab"
format_stamp: "Formatted at 2026-09-26 15:17:57 on dist-test-slave-j6q7"
server_key: "d85e366abf37a37ba467afa17c6167d6"
server_key_iv: "abbb2c74a5bf4d1afb130acac1591695"
server_key_version: "encryptionkey@0"
I20260926 15:17:57.436542 20399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:17:57.454496 20399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:17:57.457805 20399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:17:57.458072 20399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:17:57.474573 20399 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:37383
I20260926 15:17:57.474609 20465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:37383 every 8 connection(s)
I20260926 15:17:57.475673 20399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:17:57.478767 20466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:17:57.478973 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 20399
I20260926 15:17:57.479187 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:17:57.479714 6983 external_mini_cluster.cc:1428] Setting key f2741c40951d89518e4d858b564b4dfc
I20260926 15:17:57.484720 20466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37383 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.209.253@KRBTEST.COM; defaulting to no policy
Sep 26 15:17:57 dist-test-slave-j6q7 krb5kdc[20386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435877, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.253@KRBTEST.COM" created.
I20260926 15:17:57.504236 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:57.487476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:37383 (local address 127.0.0.1:53708)
0926 15:17:57.488231 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:57.488274 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:57.490068 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:57.490826 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:57.490831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:57.491500 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:17:57.491806 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:57.491819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:57.493207 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:57.493211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:57.494124 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:57.494134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:57.494891 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:57.494896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:57.495034 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:57.495598 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:17:57.495625 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:17:57.497591 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:17:57.500524 (+ 2933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:57.500537 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:57.500546 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:57.500986 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:57.501382 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:57.501386 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:57.501390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:57.501533 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:57.502179 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:17:57.502189 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:17:57.502371 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 15:17:57.502603 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 15:17:57.503084 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":144,"thread_start_us":356,"threads_started":1}
I20260926 15:17:57.504258 20470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:57.487476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53708 (local address 127.6.209.254:37383)
0926 15:17:57.489022 (+ 1546us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:57.489029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:57.489073 (+ 44us) server_negotiation.cc:407] Connection header received
0926 15:17:57.490392 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:57.490397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:57.490611 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:57.490816 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:57.491982 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:57.493065 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:57.494254 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:57.494701 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:57.494878 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:57.497783 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:57.497791 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:57.497799 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:57.497826 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:57.500388 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:57.501122 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:57.501129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:57.501131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:57.501261 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:57.501656 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:57.501662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:57.501665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:57.502054 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:57.502242 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:57.502716 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 15:17:57.503001 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1345,"spinlock_wait_cycles":19328,"thread_start_us":1133,"threads_started":1}
I20260926 15:17:57.506861 20466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:17:57.507951 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:57.507528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42813 (local address 127.0.0.1:39026)
0926 15:17:57.507705 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:57.507816 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 15:17:57.508198 20416 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42813: connect: Connection refused (error 111)
I20260926 15:17:57.509699 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:57.509253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42813 (local address 127.0.0.1:39036)
0926 15:17:57.509437 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:57.509527 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260926 15:17:57.509905 20466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.253:42813: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42813: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.253.keytab.
Entry for principal kudu/127.6.209.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.253.keytab.
I20260926 15:17:57.521982 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.253:42813
--webserver_interface=127.6.209.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.253.keytab
--principal=kudu/127.6.209.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:36477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:37383,127.6.209.253:42813,127.6.209.252:38169
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:17:57.559222 20466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } attempt: 1
I20260926 15:17:57.560343 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:57.559805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42813 (local address 127.0.0.1:39040)
0926 15:17:57.559999 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:57.560182 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 15:17:57.561288 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:57.560976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42813 (local address 127.0.0.1:39042)
0926 15:17:57.561098 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:57.561159 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 15:17:57.561501 20466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.253:42813: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42813: connect: Connection refused (error 111)
I20260926 15:17:57.629734 20466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } attempt: 2
I20260926 15:17:57.630877 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:57.630424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42813 (local address 127.0.0.1:39048)
0926 15:17:57.630670 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:57.630751 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 15:17:57.631860 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:57.631544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42813 (local address 127.0.0.1:39056)
0926 15:17:57.631671 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:57.631751 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 15:17:57.632143 20466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.253:42813: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42813: connect: Connection refused (error 111)
I20260926 15:17:57.723467 20466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } attempt: 3
I20260926 15:17:57.724531 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:57.724154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42813 (local address 127.0.0.1:39062)
0926 15:17:57.724329 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:57.724419 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 15:17:57.725474 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:57.725206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42813 (local address 127.0.0.1:39066)
0926 15:17:57.725315 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:57.725372 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 15:17:57.725732 20466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.253:42813: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42813: connect: Connection refused (error 111)
W20260926 15:17:57.807513 20473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:17:57.807760 20473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:17:57.807837 20473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:17:57.814558 20473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:17:57.814658 20473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:17:57.814697 20473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:17:57.814751 20473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:17:57.814810 20473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:17:57.823783 20473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:36477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:37383,127.6.209.253:42813,127.6.209.252:38169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.253.keytab
--principal=kudu/127.6.209.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.253:42813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.209.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:17:57.825001 20473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:17:57.826107 20473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:17:57.837064 20478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:57.837105 20481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:57.837131 20479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:17:57.837810 20473 server_base.cc:1034] running on GCE node
I20260926 15:17:57.838446 20473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:17:57.839673 20473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:17:57.840910 20473 hybrid_clock.cc:648] HybridClock initialized: now 1790435877840850 us; error 49 us; skew 500 ppm
I20260926 15:17:57.841236 20473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:17:57 dist-test-slave-j6q7 krb5kdc[20386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435877, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:17:57.843937 20473 init.cc:377] Logged in from keytab as kudu/127.6.209.253@KRBTEST.COM (short username kudu)
I20260926 15:17:57.845230 20473 webserver.cc:460] Webserver started at http://127.6.209.253:35551/ using document root <none> and password file <none>
I20260926 15:17:57.845757 20473 fs_manager.cc:359] Metadata directory not provided
I20260926 15:17:57.845844 20473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:17:57.846064 20473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:17:57.848460 20473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-1/data/instance:
uuid: "0cde5a183d734ade98b006b471be05e3"
format_stamp: "Formatted at 2026-09-26 15:17:57 on dist-test-slave-j6q7"
server_key: "3097da15feeff3e3d44419ea321fc517"
server_key_iv: "8021cb400cc3864d62c6a17d3fdf959a"
server_key_version: "encryptionkey@0"
I20260926 15:17:57.849181 20473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-1/wal/instance:
uuid: "0cde5a183d734ade98b006b471be05e3"
format_stamp: "Formatted at 2026-09-26 15:17:57 on dist-test-slave-j6q7"
server_key: "3097da15feeff3e3d44419ea321fc517"
server_key_iv: "8021cb400cc3864d62c6a17d3fdf959a"
server_key_version: "encryptionkey@0"
I20260926 15:17:57.853511 20473 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 15:17:57.856266 20488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:57.857405 20473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:17:57.857569 20473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-1/wal
uuid: "0cde5a183d734ade98b006b471be05e3"
format_stamp: "Formatted at 2026-09-26 15:17:57 on dist-test-slave-j6q7"
server_key: "3097da15feeff3e3d44419ea321fc517"
server_key_iv: "8021cb400cc3864d62c6a17d3fdf959a"
server_key_version: "encryptionkey@0"
I20260926 15:17:57.857722 20473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:17:57.868898 20466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } attempt: 4
I20260926 15:17:57.869840 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:57.869391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42813 (local address 127.0.0.1:39068)
0926 15:17:57.869631 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:57.869722 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260926 15:17:57.870823 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:57.870446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42813 (local address 127.0.0.1:39082)
0926 15:17:57.870612 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:57.870705 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 15:17:57.871054 20466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.253:42813: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42813: connect: Connection refused (error 111)
I20260926 15:17:57.886562 20473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:17:57.890098 20473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:17:57.890390 20473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:17:57.906523 20473 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.253:42813
I20260926 15:17:57.906591 20539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.253:42813 every 8 connection(s)
I20260926 15:17:57.907603 20473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-1/data/info.pb
I20260926 15:17:57.908546 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 20473
I20260926 15:17:57.908726 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-1/wal/instance
I20260926 15:17:57.909248 6983 external_mini_cluster.cc:1428] Setting key 1abdf03fd4c5d9c9fe6e33c01835ef3d
I20260926 15:17:57.911564 20540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:17:57.918815 20540 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37383 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.209.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.252@KRBTEST.COM" created.
Sep 26 15:17:57 dist-test-slave-j6q7 krb5kdc[20386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435877, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.253@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:17:57.937598 20470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:57.921098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53718 (local address 127.6.209.254:37383)
0926 15:17:57.921293 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:57.921297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:57.922455 (+ 1158us) server_negotiation.cc:407] Connection header received
0926 15:17:57.924131 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:57.924137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:57.924215 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:57.924318 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:57.925781 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:57.926463 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:57.928330 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:57.928747 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:57.928899 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:57.932392 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:57.932397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:57.932400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:57.932429 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:57.934845 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:57.935491 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:57.935498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:57.935503 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:57.935610 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:57.936008 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:57.936012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:57.936015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:57.936264 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:57.936411 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:57.937113 (+ 702us) server_negotiation.cc:299] Negotiation successful
0926 15:17:57.937309 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 15:17:57.938364 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:57.921659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:37383 (local address 127.0.0.1:53718)
0926 15:17:57.922311 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:57.922342 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:57.923883 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:57.924536 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:57.924543 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:57.925118 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:17:57.925615 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:57.925630 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:57.926636 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:57.926640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:57.928199 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:57.928215 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:57.928982 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:57.928989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:57.929167 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:57.929745 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:17:57.929764 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:17:57.932197 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:17:57.934978 (+ 2781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:57.934988 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:57.935000 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:57.935367 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:57.935729 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:57.935735 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:57.935738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:57.935887 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:57.936392 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:17:57.936398 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:17:57.936688 (+ 290us) client_negotiation.cc:769] Sending connection context
0926 15:17:57.937010 (+ 322us) client_negotiation.cc:240] Negotiation successful
0926 15:17:57.937403 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":272,"threads_started":1}
I20260926 15:17:57.939870 20540 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:17:57 dist-test-slave-j6q7 krb5kdc[20386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435877, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.253@KRBTEST.COM for kudu/127.6.209.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.252.keytab.
Entry for principal kudu/127.6.209.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.252.keytab.
I20260926 15:17:57.955741 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:57.940716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42813 (local address 127.0.0.1:39094)
0926 15:17:57.940922 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:57.940941 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:57.941034 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:57.942005 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:57.942008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:57.942283 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:17:57.942390 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:57.942396 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:57.944273 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:57.944276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:57.945089 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:57.945099 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:57.946142 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:57.946145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:57.946284 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:57.946755 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:17:57.946768 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:17:57.950616 (+ 3848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:17:57.953748 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:57.953751 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:57.953753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:57.954072 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:57.954387 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:57.954389 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:57.954391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:57.954502 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:57.955059 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:17:57.955064 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:17:57.955189 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 15:17:57.955316 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 15:17:57.955508 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":1}
I20260926 15:17:57.956120 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:57.940848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39094 (local address 127.6.209.253:42813)
0926 15:17:57.941554 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:57.941564 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:57.941595 (+ 31us) server_negotiation.cc:407] Connection header received
0926 15:17:57.941648 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:57.941658 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:57.941866 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:57.942009 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:57.942532 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:57.944148 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:57.945214 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:57.946023 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:57.946135 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:57.950799 (+ 4664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:57.950810 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:57.950820 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:57.950846 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:57.953639 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:57.954187 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:57.954199 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:57.954201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:57.954302 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:57.954604 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:57.954608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:57.954611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:57.954899 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:57.955087 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:57.955489 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 15:17:57.955735 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":518,"thread_start_us":398,"threads_started":1}
I20260926 15:17:57.956334 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.252:38169
--webserver_interface=127.6.209.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.252.keytab
--principal=kudu/127.6.209.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:36477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:37383,127.6.209.253:42813,127.6.209.252:38169
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:17:57.958163 20540 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:17:57.959085 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:57.958705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:38169 (local address 127.0.0.1:57012)
0926 15:17:57.958848 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:57.958965 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 15:17:57.959394 20490 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
I20260926 15:17:57.961087 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:57.960592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:38169 (local address 127.0.0.1:57018)
0926 15:17:57.960791 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:57.960918 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 15:17:57.961369 20540 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.252:38169: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
I20260926 15:17:58.010762 20540 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } attempt: 1
I20260926 15:17:58.011885 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.011438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:38169 (local address 127.0.0.1:57030)
0926 15:17:58.011632 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:58.011751 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 15:17:58.012972 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.012575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:38169 (local address 127.0.0.1:57044)
0926 15:17:58.012723 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:58.012787 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 15:17:58.013247 20540 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.252:38169: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
I20260926 15:17:58.081491 20540 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } attempt: 2
I20260926 15:17:58.082557 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.082150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:38169 (local address 127.0.0.1:57046)
0926 15:17:58.082329 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:58.082412 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 15:17:58.083468 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.083150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:38169 (local address 127.0.0.1:57062)
0926 15:17:58.083288 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:58.083353 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 15:17:58.083667 20540 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.252:38169: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
I20260926 15:17:58.170310 20466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } attempt: 5
I20260926 15:17:58.174888 20540 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } attempt: 3
I20260926 15:17:58.175798 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.175350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:38169 (local address 127.0.0.1:57064)
0926 15:17:58.175581 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:58.175654 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 15:17:58.177076 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.176555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:38169 (local address 127.0.0.1:57076)
0926 15:17:58.176803 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:58.176922 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
W20260926 15:17:58.177366 20540 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.252:38169: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
Sep 26 15:17:58 dist-test-slave-j6q7 krb5kdc[20386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435877, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for kudu/127.6.209.253@KRBTEST.COM
I20260926 15:17:58.183399 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.171264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39106 (local address 127.6.209.253:42813)
0926 15:17:58.171478 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:58.171484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:58.171507 (+ 23us) server_negotiation.cc:407] Connection header received
0926 15:17:58.171572 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:58.171575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:58.171631 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:58.171753 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:58.172460 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.173185 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.174263 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.174646 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.174777 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:58.178489 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:58.178494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:58.178497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:58.178523 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:58.181113 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:58.181749 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:58.181754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:58.181757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:58.181862 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:58.182194 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:58.182199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:58.182202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:58.182429 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:58.182569 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:58.182939 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 15:17:58.183128 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 15:17:58.183432 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.171027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42813 (local address 127.0.0.1:39106)
0926 15:17:58.171215 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:58.171239 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:58.171398 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:58.171789 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:58.171796 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:58.172156 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:17:58.172312 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:58.172320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.173319 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.173326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:58.174121 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:58.174131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.174806 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.174812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:58.174971 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:58.176072 (+ 1101us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:17:58.176097 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:17:58.178319 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:17:58.181247 (+ 2928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:58.181252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:58.181256 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:58.181609 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:58.181962 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:58.181967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:58.181977 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:58.182090 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:58.182538 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:17:58.182542 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:17:58.182683 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 15:17:58.182847 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 15:17:58.183127 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 15:17:58.184465 20466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:17:58.185314 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.185001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:38169 (local address 127.0.0.1:57082)
0926 15:17:58.185136 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:58.185204 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 15:17:58.186064 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.185801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:38169 (local address 127.0.0.1:57094)
0926 15:17:58.185931 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:58.185986 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 15:17:58.186261 20466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.252:38169: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
I20260926 15:17:58.237466 20466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } attempt: 1
I20260926 15:17:58.238627 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.238051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:38169 (local address 127.0.0.1:57104)
0926 15:17:58.238287 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:58.238413 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 15:17:58.239807 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.239375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:38169 (local address 127.0.0.1:57120)
0926 15:17:58.239530 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:58.239628 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 15:17:58.240039 20466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.252:38169: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
W20260926 15:17:58.250885 20547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:17:58.251137 20547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:17:58.251233 20547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:17:58.258383 20547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:17:58.258481 20547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:17:58.258541 20547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:17:58.258596 20547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:17:58.258659 20547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:17:58.268254 20547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:36477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:37383,127.6.209.253:42813,127.6.209.252:38169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.252.keytab
--principal=kudu/127.6.209.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.252:38169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.209.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:17:58.269434 20547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:17:58.270622 20547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:17:58.281847 20552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:58.282007 20555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:17:58.282071 20553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:17:58.283092 20547 server_base.cc:1034] running on GCE node
I20260926 15:17:58.283767 20547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:17:58.285013 20547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:17:58.286284 20547 hybrid_clock.cc:648] HybridClock initialized: now 1790435878286226 us; error 59 us; skew 500 ppm
I20260926 15:17:58.286602 20547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:17:58 dist-test-slave-j6q7 krb5kdc[20386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435878, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:17:58.289503 20547 init.cc:377] Logged in from keytab as kudu/127.6.209.252@KRBTEST.COM (short username kudu)
I20260926 15:17:58.290819 20547 webserver.cc:460] Webserver started at http://127.6.209.252:38765/ using document root <none> and password file <none>
I20260926 15:17:58.291432 20547 fs_manager.cc:359] Metadata directory not provided
I20260926 15:17:58.291576 20547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:17:58.291853 20547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:17:58.294534 20547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-2/data/instance:
uuid: "cd7f6988da5b4c66b14db69262673f07"
format_stamp: "Formatted at 2026-09-26 15:17:58 on dist-test-slave-j6q7"
server_key: "e34ca3dec59b9d065f32a2664a77af5d"
server_key_iv: "9bf315c431f16f0239ba8d25e7733aaa"
server_key_version: "encryptionkey@0"
I20260926 15:17:58.295284 20547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-2/wal/instance:
uuid: "cd7f6988da5b4c66b14db69262673f07"
format_stamp: "Formatted at 2026-09-26 15:17:58 on dist-test-slave-j6q7"
server_key: "e34ca3dec59b9d065f32a2664a77af5d"
server_key_iv: "9bf315c431f16f0239ba8d25e7733aaa"
server_key_version: "encryptionkey@0"
I20260926 15:17:58.299819 20547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 15:17:58.302913 20562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:17:58.304133 20547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 15:17:58.304260 20547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-2/wal
uuid: "cd7f6988da5b4c66b14db69262673f07"
format_stamp: "Formatted at 2026-09-26 15:17:58 on dist-test-slave-j6q7"
server_key: "e34ca3dec59b9d065f32a2664a77af5d"
server_key_iv: "9bf315c431f16f0239ba8d25e7733aaa"
server_key_version: "encryptionkey@0"
I20260926 15:17:58.304399 20547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:17:58.308272 20466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } attempt: 2
I20260926 15:17:58.309453 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.308909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:38169 (local address 127.0.0.1:57134)
0926 15:17:58.309168 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:58.309276 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 15:17:58.310385 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.310059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:38169 (local address 127.0.0.1:57144)
0926 15:17:58.310210 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:58.310292 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 15:17:58.310635 20466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.252:38169: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
I20260926 15:17:58.320582 20540 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } attempt: 4
I20260926 15:17:58.321597 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.321065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:38169 (local address 127.0.0.1:57152)
0926 15:17:58.321291 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:58.321389 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 15:17:58.322599 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.322193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:38169 (local address 127.0.0.1:57162)
0926 15:17:58.322385 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:58.322479 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 15:17:58.322829 20540 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.252:38169: Network error: Client connection negotiation failed: client connection to 127.6.209.252:38169: connect: Connection refused (error 111)
I20260926 15:17:58.323423 20547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:17:58.327198 20547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:17:58.327456 20547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:17:58.345082 20547 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.252:38169
I20260926 15:17:58.345108 20613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.252:38169 every 8 connection(s)
I20260926 15:17:58.346221 20547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-2/data/info.pb
I20260926 15:17:58.349190 20614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:17:58.354426 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 20547
I20260926 15:17:58.354610 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435694426955-6983-0/minicluster-data/master-2/wal/instance
I20260926 15:17:58.355161 6983 external_mini_cluster.cc:1428] Setting key c96689f4efb1b72c7518884c605d8577
I20260926 15:17:58.355376 20614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37383 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:17:58 dist-test-slave-j6q7 krb5kdc[20386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435878, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.252@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
Sep 26 15:17:58 dist-test-slave-j6q7 krb5kdc[20386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.252@KRBTEST.COM
I20260926 15:17:58.374298 20470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.357762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53720 (local address 127.6.209.254:37383)
0926 15:17:58.357934 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:58.357939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:58.358647 (+ 708us) server_negotiation.cc:407] Connection header received
0926 15:17:58.359271 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:58.359277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:58.359344 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:58.359439 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:58.361060 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.362181 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.364057 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.364570 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.364699 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:58.368466 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:58.368471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:58.368474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:58.368509 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:58.371233 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:58.372239 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:58.372243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:58.372245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:58.372355 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:58.372743 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:58.372749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:58.372752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:58.373041 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:58.373189 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:58.373798 (+ 609us) server_negotiation.cc:299] Negotiation successful
0926 15:17:58.373984 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 15:17:58.375146 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.356742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57172 (local address 127.6.209.252:38169)
0926 15:17:58.357394 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:58.357403 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:58.357434 (+ 31us) server_negotiation.cc:407] Connection header received
0926 15:17:58.359054 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:58.359072 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:58.359304 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:58.359488 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:58.361220 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.362721 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.364339 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.365230 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.365314 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:58.369341 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:58.369348 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:58.369357 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:58.369383 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:58.372333 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:58.372913 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:58.372920 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:58.372923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:58.373037 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:58.373393 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:58.373398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:58.373401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:58.373715 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:58.373987 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:58.374305 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 15:17:58.374532 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":507,"thread_start_us":295,"threads_started":1}
I20260926 15:17:58.375178 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.357782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:37383 (local address 127.0.0.1:53720)
0926 15:17:58.358503 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:58.358541 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:58.359062 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:58.359501 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:58.359506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:58.360288 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:17:58.360864 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:58.360890 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.362329 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.362333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:58.363913 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:58.363927 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.364776 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.364781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:58.365117 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:58.365754 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:17:58.365783 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:17:58.368262 (+ 2479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:17:58.371369 (+ 3107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:58.371380 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:58.371390 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:58.372104 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:58.372462 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:58.372467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:58.372471 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:58.372639 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:58.373171 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:17:58.373178 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:17:58.373449 (+ 271us) client_negotiation.cc:769] Sending connection context
0926 15:17:58.373714 (+ 265us) client_negotiation.cc:240] Negotiation successful
0926 15:17:58.374120 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":20,"thread_start_us":365,"threads_started":1}
I20260926 15:17:58.376366 20614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:17:58 dist-test-slave-j6q7 krb5kdc[20386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435878, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.252@KRBTEST.COM for kudu/127.6.209.253@KRBTEST.COM
Sep 26 15:17:58 dist-test-slave-j6q7 krb5kdc[20386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:17:58.390154 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.377193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42813 (local address 127.0.0.1:39118)
0926 15:17:58.377380 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:58.377396 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:58.377497 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:58.377839 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:58.377845 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:58.378179 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:17:58.378308 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:58.378315 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.379302 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.379307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:58.380542 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:58.380555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.381403 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.381409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:58.381614 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:58.382164 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:17:58.382179 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:17:58.384265 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:17:58.387649 (+ 3384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:58.387657 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:58.387661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:58.388040 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:58.388461 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:58.388468 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:58.388472 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:58.388631 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:58.389201 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:17:58.389206 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:17:58.389361 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 15:17:58.389508 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 15:17:58.389766 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 15:17:58.390200 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.377054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39118 (local address 127.6.209.253:42813)
0926 15:17:58.377242 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:58.377248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:58.377456 (+ 208us) server_negotiation.cc:407] Connection header received
0926 15:17:58.377610 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:58.377613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:58.377707 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:58.377802 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:58.378433 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.379147 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.380708 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.381239 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.381367 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:58.384468 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:58.384475 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:58.384478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:58.384513 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:58.387512 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:58.388168 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:58.388176 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:58.388180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:58.388342 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:58.388746 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:58.388753 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:58.388757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:58.389075 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:58.389244 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:58.389596 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 15:17:58.389799 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 15:17:58.391512 20470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.377942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53724 (local address 127.6.209.254:37383)
0926 15:17:58.378098 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:58.378104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:58.378175 (+ 71us) server_negotiation.cc:407] Connection header received
0926 15:17:58.378354 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:58.378359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:58.378423 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:58.378516 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:58.379647 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.380365 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.381972 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.382502 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.382607 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:58.386591 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:58.386597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:58.386601 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:58.386636 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:58.389138 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:58.389759 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:58.389765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:58.389768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:58.389900 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:58.390257 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:58.390263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:58.390266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:58.390503 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:58.390782 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:58.391012 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 15:17:58.391209 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:17:58.391492 20614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:17:58 dist-test-slave-j6q7 krb5kdc[20386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435878, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.252@KRBTEST.COM for kudu/127.6.209.252@KRBTEST.COM
Sep 26 15:17:58 dist-test-slave-j6q7 krb5kdc[20386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.253@KRBTEST.COM
I20260926 15:17:58.405808 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.392403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57180 (local address 127.6.209.252:38169)
0926 15:17:58.392639 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:58.392643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:58.392658 (+ 15us) server_negotiation.cc:407] Connection header received
0926 15:17:58.392694 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:58.392697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:58.392777 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:58.392911 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:58.393695 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.394352 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.395573 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.396022 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.396125 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:58.399558 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:58.399568 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:58.399572 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:58.399621 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:58.403147 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:58.403787 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:58.403793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:58.403797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:58.403994 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:58.404361 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:58.404367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:58.404371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:58.404743 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:58.404957 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:58.405462 (+ 505us) server_negotiation.cc:299] Negotiation successful
0926 15:17:58.405608 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260926 15:17:58.406349 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.392141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:38169 (local address 127.0.0.1:57180)
0926 15:17:58.392300 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:58.392316 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:58.392401 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:58.392935 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:58.392939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:58.393400 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:17:58.393584 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:58.393595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.394482 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.394486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:58.395434 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:58.395444 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.396165 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.396171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:58.396372 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:58.396882 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:17:58.396902 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:17:58.399313 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:17:58.403280 (+ 3967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:58.403283 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:58.403285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:58.403646 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:58.404123 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:58.404126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:58.404128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:58.404247 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:58.405013 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:17:58.405018 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:17:58.405184 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 15:17:58.405372 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 15:17:58.405796 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":107}
I20260926 15:17:58.409256 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.393067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39128 (local address 127.6.209.253:42813)
0926 15:17:58.393309 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:58.393315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:58.393331 (+ 16us) server_negotiation.cc:407] Connection header received
0926 15:17:58.393432 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:58.393436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:58.393483 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:58.393560 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:58.394795 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.395466 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.397000 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.397403 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.397504 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:58.403068 (+ 5564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:58.403078 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:58.403084 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:58.403122 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:58.406523 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:58.407306 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:58.407312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:58.407318 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:58.407501 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:58.407902 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:58.407907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:58.407910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:58.408147 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:58.408413 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:58.408743 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 15:17:58.408981 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260926 15:17:58.411546 20614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07: Bootstrap starting.
I20260926 15:17:58.413609 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:17:58.414857 20614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07: Neither blocks nor log segments found. Creating new log.
I20260926 15:17:58.415795 20614 log.cc:824] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07: Log is configured to *not* fsync() on all Append() calls
I20260926 15:17:58.416855 20466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } attempt: 3
I20260926 15:17:58.419232 20614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07: No bootstrap required, opened a new log
Sep 26 15:17:58 dist-test-slave-j6q7 krb5kdc[20386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435877, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for kudu/127.6.209.252@KRBTEST.COM
I20260926 15:17:58.433010 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.417334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:38169 (local address 127.0.0.1:57196)
0926 15:17:58.417496 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:58.417513 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:58.417607 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:58.417988 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:58.417994 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:58.418325 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:17:58.418474 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:58.418481 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.419465 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.419470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:58.420366 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:58.420376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.421331 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.421344 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:58.421528 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:58.422076 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:17:58.422105 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:17:58.425290 (+ 3185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:17:58.430133 (+ 4843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:58.430139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:58.430143 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:58.430613 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:58.431073 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:58.431077 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:58.431081 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:58.431251 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:58.431993 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:17:58.431998 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:17:58.432199 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 15:17:58.432320 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 15:17:58.432597 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 15:17:58.433344 20614 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3005514e2a9e44a8ac3bfe4815a909ab" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37383 } } peers { permanent_uuid: "0cde5a183d734ade98b006b471be05e3" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } } peers { permanent_uuid: "cd7f6988da5b4c66b14db69262673f07" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } }
I20260926 15:17:58.433828 20614 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:17:58.434026 20614 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7f6988da5b4c66b14db69262673f07, State: Initialized, Role: FOLLOWER
I20260926 15:17:58.434382 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.417563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57196 (local address 127.6.209.252:38169)
0926 15:17:58.417751 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:58.417756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:58.417777 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:17:58.417809 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:58.417812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:58.417863 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:58.417961 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:58.418604 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.419277 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.420502 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.421185 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.421322 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:58.425558 (+ 4236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:58.425567 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:58.425574 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:58.425619 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:58.429991 (+ 4372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:58.430762 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:58.430768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:58.430774 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:58.430929 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:58.431434 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:58.431441 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:58.431445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:58.431835 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:58.433730 (+ 1895us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:58.433844 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 15:17:58.434040 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 15:17:58.435127 20614 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3005514e2a9e44a8ac3bfe4815a909ab" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37383 } } peers { permanent_uuid: "0cde5a183d734ade98b006b471be05e3" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } } peers { permanent_uuid: "cd7f6988da5b4c66b14db69262673f07" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } }
I20260926 15:17:58.440109 20466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3005514e2a9e44a8ac3bfe4815a909ab: Bootstrap starting.
I20260926 15:17:58.444044 20466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3005514e2a9e44a8ac3bfe4815a909ab: Neither blocks nor log segments found. Creating new log.
I20260926 15:17:58.445554 20466 log.cc:824] T 00000000000000000000000000000000 P 3005514e2a9e44a8ac3bfe4815a909ab: Log is configured to *not* fsync() on all Append() calls
I20260926 15:17:58.446980 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.422754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39136 (local address 127.6.209.253:42813)
0926 15:17:58.428070 (+ 5316us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:58.428076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:58.429949 (+ 1873us) server_negotiation.cc:407] Connection header received
0926 15:17:58.430336 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:58.430342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:58.430413 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:58.430480 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:58.433266 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.434289 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.436103 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.436685 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.436773 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:58.440020 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:58.440026 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:58.440030 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:58.440084 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:58.443471 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:58.444335 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:58.444340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:58.444344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:58.444497 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:58.444941 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:58.444946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:58.444950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:58.445280 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:58.445512 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:58.446065 (+ 553us) server_negotiation.cc:299] Negotiation successful
0926 15:17:58.446339 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5198}
I20260926 15:17:58.454761 20466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3005514e2a9e44a8ac3bfe4815a909ab: No bootstrap required, opened a new log
I20260926 15:17:58.462303 20627 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3005514e2a9e44a8ac3bfe4815a909ab" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37383 } } peers { permanent_uuid: "0cde5a183d734ade98b006b471be05e3" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } } peers { permanent_uuid: "cd7f6988da5b4c66b14db69262673f07" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } } }
I20260926 15:17:58.464923 20627 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:17:58.463542 20614 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:17:58.467041 20470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.422369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53726 (local address 127.6.209.254:37383)
0926 15:17:58.422544 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:58.422549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:58.433528 (+ 10979us) server_negotiation.cc:407] Connection header received
0926 15:17:58.447374 (+ 13846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:58.447384 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:58.447494 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:58.447588 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:58.450363 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.451782 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.454289 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.455006 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.455115 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:58.459042 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:58.459050 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:58.459056 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:58.459104 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:58.463421 (+ 4317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:58.464408 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:58.464414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:58.464418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:58.464598 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:58.465083 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:58.465089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:58.465092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:58.465523 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:58.465791 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:58.466402 (+ 611us) server_negotiation.cc:299] Negotiation successful
0926 15:17:58.466628 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:17:58.467068 20466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3005514e2a9e44a8ac3bfe4815a909ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3005514e2a9e44a8ac3bfe4815a909ab" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37383 } } peers { permanent_uuid: "0cde5a183d734ade98b006b471be05e3" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } } peers { permanent_uuid: "cd7f6988da5b4c66b14db69262673f07" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } }
I20260926 15:17:58.467453 20466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3005514e2a9e44a8ac3bfe4815a909ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:17:58.467574 20466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3005514e2a9e44a8ac3bfe4815a909ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3005514e2a9e44a8ac3bfe4815a909ab, State: Initialized, Role: FOLLOWER
I20260926 15:17:58.468195 20466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3005514e2a9e44a8ac3bfe4815a909ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3005514e2a9e44a8ac3bfe4815a909ab" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37383 } } peers { permanent_uuid: "0cde5a183d734ade98b006b471be05e3" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } } peers { permanent_uuid: "cd7f6988da5b4c66b14db69262673f07" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } }
I20260926 15:17:58.473083 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.433946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57204 (local address 127.6.209.252:38169)
0926 15:17:58.444975 (+ 11029us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:58.444982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:58.446549 (+ 1567us) server_negotiation.cc:407] Connection header received
0926 15:17:58.450952 (+ 4403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:58.450961 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:58.451051 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:58.451132 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:58.455336 (+ 4204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.456746 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.459438 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.460416 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.460524 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:58.464896 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:58.464902 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:58.464913 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:58.464949 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:58.469134 (+ 4185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:58.470218 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:58.470228 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:58.470234 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:58.470431 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:58.470945 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:58.470953 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:58.470959 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:58.471456 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:58.471783 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:58.472134 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 15:17:58.472443 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10939,"thread_start_us":21213,"threads_started":1}
I20260926 15:17:58.474735 20466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3005514e2a9e44a8ac3bfe4815a909ab [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:17:58.473356 20631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3005514e2a9e44a8ac3bfe4815a909ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3005514e2a9e44a8ac3bfe4815a909ab" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37383 } } peers { permanent_uuid: "0cde5a183d734ade98b006b471be05e3" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } } peers { permanent_uuid: "cd7f6988da5b4c66b14db69262673f07" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } } }
I20260926 15:17:58.475003 20631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3005514e2a9e44a8ac3bfe4815a909ab [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:17:58.485265 20547 master_runner.cc:186] Error getting master registration for 127.6.209.254:37383: OK, instance_id { permanent_uuid: "3005514e2a9e44a8ac3bfe4815a909ab" instance_seqno: 1790435877469908 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 15:17:58.497033 20547 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 15:17:58.502431 20643 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:17:58.502549 20643 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 15:17:58.508648 20653 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3005514e2a9e44a8ac3bfe4815a909ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:17:58.508831 20653 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:17:58.507774 20399 master_runner.cc:186] Error getting master registration for 127.6.209.253:42813: OK, instance_id { permanent_uuid: "0cde5a183d734ade98b006b471be05e3" instance_seqno: 1790435877902030 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 15:17:58.518564 20399 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 15:17:58.579361 20568 tablet.cc:2369] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:17:58.612565 20627 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:17:58.612869 20627 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3005514e2a9e44a8ac3bfe4815a909ab" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37383 } } peers { permanent_uuid: "0cde5a183d734ade98b006b471be05e3" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } } peers { permanent_uuid: "cd7f6988da5b4c66b14db69262673f07" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } }
I20260926 15:17:58.614475 20627 leader_election.cc:288] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3005514e2a9e44a8ac3bfe4815a909ab (127.6.209.254:37383), 0cde5a183d734ade98b006b471be05e3 (127.6.209.253:42813)
I20260926 15:17:58.615087 20441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd7f6988da5b4c66b14db69262673f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3005514e2a9e44a8ac3bfe4815a909ab" is_pre_election: true
I20260926 15:17:58.615612 20441 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3005514e2a9e44a8ac3bfe4815a909ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd7f6988da5b4c66b14db69262673f07 in term 0.
I20260926 15:17:58.617146 20565 leader_election.cc:302] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3005514e2a9e44a8ac3bfe4815a909ab, cd7f6988da5b4c66b14db69262673f07; no voters:
I20260926 15:17:58.617520 20627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:17:58.617652 20627 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:17:58.617736 20627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:17:58.620779 20627 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3005514e2a9e44a8ac3bfe4815a909ab" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37383 } } peers { permanent_uuid: "0cde5a183d734ade98b006b471be05e3" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } } peers { permanent_uuid: "cd7f6988da5b4c66b14db69262673f07" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } }
I20260926 15:17:58.621829 20441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd7f6988da5b4c66b14db69262673f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3005514e2a9e44a8ac3bfe4815a909ab"
I20260926 15:17:58.621907 20627 leader_election.cc:288] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [CANDIDATE]: Term 1 election: Requested vote from peers 3005514e2a9e44a8ac3bfe4815a909ab (127.6.209.254:37383), 0cde5a183d734ade98b006b471be05e3 (127.6.209.253:42813)
I20260926 15:17:58.622038 20441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3005514e2a9e44a8ac3bfe4815a909ab [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:17:58.624886 20540 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } attempt: 5
I20260926 15:17:58.628752 20441 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3005514e2a9e44a8ac3bfe4815a909ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd7f6988da5b4c66b14db69262673f07 in term 1.
I20260926 15:17:58.629393 20565 leader_election.cc:302] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3005514e2a9e44a8ac3bfe4815a909ab, cd7f6988da5b4c66b14db69262673f07; no voters:
I20260926 15:17:58.630103 20627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [term 1 FOLLOWER]: Leader election won for term 1
Sep 26 15:17:58 dist-test-slave-j6q7 krb5kdc[20386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435877, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.253@KRBTEST.COM for kudu/127.6.209.252@KRBTEST.COM
I20260926 15:17:58.635375 20627 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [term 1 LEADER]: Becoming Leader. State: Replica: cd7f6988da5b4c66b14db69262673f07, State: Running, Role: LEADER
I20260926 15:17:58.636229 20627 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3005514e2a9e44a8ac3bfe4815a909ab" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37383 } } peers { permanent_uuid: "0cde5a183d734ade98b006b471be05e3" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } } peers { permanent_uuid: "cd7f6988da5b4c66b14db69262673f07" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } }
I20260926 15:17:58.650251 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.625662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57208 (local address 127.6.209.252:38169)
0926 15:17:58.625804 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 15:17:58.625810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:17:58.625826 (+ 16us) server_negotiation.cc:407] Connection header received
0926 15:17:58.625873 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:17:58.625878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:17:58.625960 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:17:58.626046 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:17:58.626910 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.627964 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.629738 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.630378 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.630480 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:58.636270 (+ 5790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:17:58.636276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:17:58.636280 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:17:58.636336 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:17:58.641041 (+ 4705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:58.642245 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:58.642251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:58.642255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:58.642496 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:17:58.643169 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:17:58.643174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:17:58.643177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:17:58.643688 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:17:58.643881 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 15:17:58.648674 (+ 4793us) server_negotiation.cc:299] Negotiation successful
0926 15:17:58.649092 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 15:17:58.650882 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:17:58.625463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:38169 (local address 127.0.0.1:57208)
0926 15:17:58.625628 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 15:17:58.625646 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:17:58.625743 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:17:58.626069 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:17:58.626073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:17:58.626598 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:17:58.626781 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:58.626792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.628088 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.628093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:58.629575 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:17:58.629588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:17:58.630725 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:17:58.630732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:17:58.630962 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:17:58.631677 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:17:58.631701 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:17:58.635988 (+ 4287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:17:58.641222 (+ 5234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:58.641229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:58.641234 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:58.642082 (+ 848us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:58.642731 (+ 649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:17:58.642735 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:17:58.642738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:17:58.642982 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:17:58.644004 (+ 1022us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:17:58.644008 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:17:58.644200 (+ 192us) client_negotiation.cc:769] Sending connection context
0926 15:17:58.649035 (+ 4835us) client_negotiation.cc:240] Negotiation successful
0926 15:17:58.649803 (+ 768us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 15:17:58.655540 20655 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd7f6988da5b4c66b14db69262673f07. Latest consensus state: current_term: 1 leader_uuid: "cd7f6988da5b4c66b14db69262673f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3005514e2a9e44a8ac3bfe4815a909ab" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37383 } } peers { permanent_uuid: "0cde5a183d734ade98b006b471be05e3" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } } peers { permanent_uuid: "cd7f6988da5b4c66b14db69262673f07" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } } }
I20260926 15:17:58.656051 20655 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [sys.catalog]: This master's current role is: LEADER
I20260926 15:17:58.656082 20540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cde5a183d734ade98b006b471be05e3: Bootstrap starting.
I20260926 15:17:58.664414 20540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cde5a183d734ade98b006b471be05e3: Neither blocks nor log segments found. Creating new log.
I20260926 15:17:58.666218 20540 log.cc:824] T 00000000000000000000000000000000 P 0cde5a183d734ade98b006b471be05e3: Log is configured to *not* fsync() on all Append() calls
I20260926 15:17:58.669350 20657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:17:58.672636 20657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:17:58.677716 20540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cde5a183d734ade98b006b471be05e3: No bootstrap required, opened a new log
I20260926 15:17:58.693207 20441 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3005514e2a9e44a8ac3bfe4815a909ab [term 1 FOLLOWER]: Refusing update from remote peer cd7f6988da5b4c66b14db69262673f07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:17:58.694234 20655 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3005514e2a9e44a8ac3bfe4815a909ab" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:17:58.698944 20540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cde5a183d734ade98b006b471be05e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3005514e2a9e44a8ac3bfe4815a909ab" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37383 } } peers { permanent_uuid: "0cde5a183d734ade98b006b471be05e3" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } } peers { permanent_uuid: "cd7f6988da5b4c66b14db69262673f07" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } }
I20260926 15:17:58.699613 20540 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cde5a183d734ade98b006b471be05e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:17:58.699875 20540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cde5a183d734ade98b006b471be05e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cde5a183d734ade98b006b471be05e3, State: Initialized, Role: FOLLOWER
I20260926 15:17:58.704146 20540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cde5a183d734ade98b006b471be05e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3005514e2a9e44a8ac3bfe4815a909ab" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37383 } } peers { permanent_uuid: "0cde5a183d734ade98b006b471be05e3" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } } peers { permanent_uuid: "cd7f6988da5b4c66b14db69262673f07" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } }
W20260926 15:17:58.710448 20420 tablet.cc:2369] T 00000000000000000000000000000000 P 3005514e2a9e44a8ac3bfe4815a909ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:17:58.721316 20661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cde5a183d734ade98b006b471be05e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3005514e2a9e44a8ac3bfe4815a909ab" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37383 } } peers { permanent_uuid: "0cde5a183d734ade98b006b471be05e3" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } } peers { permanent_uuid: "cd7f6988da5b4c66b14db69262673f07" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } } }
I20260926 15:17:58.721763 20661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cde5a183d734ade98b006b471be05e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:17:58.723662 20514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd7f6988da5b4c66b14db69262673f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cde5a183d734ade98b006b471be05e3"
I20260926 15:17:58.724056 20514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cde5a183d734ade98b006b471be05e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:17:58.727246 20514 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0cde5a183d734ade98b006b471be05e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd7f6988da5b4c66b14db69262673f07 in term 1.
I20260926 15:17:58.728032 20513 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0cde5a183d734ade98b006b471be05e3 [term 1 FOLLOWER]: Refusing update from remote peer cd7f6988da5b4c66b14db69262673f07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:17:58.728567 20515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd7f6988da5b4c66b14db69262673f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cde5a183d734ade98b006b471be05e3" is_pre_election: true
I20260926 15:17:58.728880 20515 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 0cde5a183d734ade98b006b471be05e3 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cd7f6988da5b4c66b14db69262673f07 in term 1. Re-sending same reply.
I20260926 15:17:58.728794 20655 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cde5a183d734ade98b006b471be05e3" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:17:58.729206 20540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cde5a183d734ade98b006b471be05e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:17:58.736642 20631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3005514e2a9e44a8ac3bfe4815a909ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd7f6988da5b4c66b14db69262673f07. Latest consensus state: current_term: 1 leader_uuid: "cd7f6988da5b4c66b14db69262673f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3005514e2a9e44a8ac3bfe4815a909ab" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37383 } } peers { permanent_uuid: "0cde5a183d734ade98b006b471be05e3" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } } peers { permanent_uuid: "cd7f6988da5b4c66b14db69262673f07" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } } }
I20260926 15:17:58.736943 20631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3005514e2a9e44a8ac3bfe4815a909ab [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:17:58.738919 20659 mvcc.cc:205] Tried to move back new op lower bound from 7333625359111286784 to 7333625358902272000. Current Snapshot: MvccSnapshot[applied={T|T < 7333625359111286784}]
I20260926 15:17:58.740778 20660 mvcc.cc:205] Tried to move back new op lower bound from 7333625359111286784 to 7333625358902272000. Current Snapshot: MvccSnapshot[applied={T|T < 7333625359111286784}]
I20260926 15:17:58.753048 20627 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd7f6988da5b4c66b14db69262673f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3005514e2a9e44a8ac3bfe4815a909ab" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37383 } } peers { permanent_uuid: "0cde5a183d734ade98b006b471be05e3" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } } peers { permanent_uuid: "cd7f6988da5b4c66b14db69262673f07" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } } }
I20260926 15:17:58.753300 20627 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [sys.catalog]: This master's current role is: LEADER
I20260926 15:17:58.756228 20661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cde5a183d734ade98b006b471be05e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd7f6988da5b4c66b14db69262673f07. Latest consensus state: current_term: 1 leader_uuid: "cd7f6988da5b4c66b14db69262673f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3005514e2a9e44a8ac3bfe4815a909ab" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37383 } } peers { permanent_uuid: "0cde5a183d734ade98b006b471be05e3" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } } peers { permanent_uuid: "cd7f6988da5b4c66b14db69262673f07" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } } }
I20260926 15:17:58.756501 20661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cde5a183d734ade98b006b471be05e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:17:58.759126 20631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3005514e2a9e44a8ac3bfe4815a909ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd7f6988da5b4c66b14db69262673f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3005514e2a9e44a8ac3bfe4815a909ab" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37383 } } peers { permanent_uuid: "0cde5a183d734ade98b006b471be05e3" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } } peers { permanent_uuid: "cd7f6988da5b4c66b14db69262673f07" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } } }
I20260926 15:17:58.759428 20631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3005514e2a9e44a8ac3bfe4815a909ab [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:17:58.760835 20664 mvcc.cc:205] Tried to move back new op lower bound from 7333625359111286784 to 7333625358902272000. Current Snapshot: MvccSnapshot[applied={T|T < 7333625359111286784}]
I20260926 15:17:58.762823 20657 catalog_manager.cc:1347] Generated new cluster ID: 35f9eac82b6a4e079b59634488351d3e
I20260926 15:17:58.762924 20657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:17:58.785224 20655 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd7f6988da5b4c66b14db69262673f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3005514e2a9e44a8ac3bfe4815a909ab" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37383 } } peers { permanent_uuid: "0cde5a183d734ade98b006b471be05e3" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } } peers { permanent_uuid: "cd7f6988da5b4c66b14db69262673f07" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } } }
I20260926 15:17:58.785481 20655 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07 [sys.catalog]: This master's current role is: LEADER
I20260926 15:17:58.805727 20661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cde5a183d734ade98b006b471be05e3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd7f6988da5b4c66b14db69262673f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3005514e2a9e44a8ac3bfe4815a909ab" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 37383 } } peers { permanent_uuid: "0cde5a183d734ade98b006b471be05e3" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42813 } } peers { permanent_uuid: "cd7f6988da5b4c66b14db69262673f07" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 38169 } } }
I20260926 15:17:58.805980 20661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cde5a183d734ade98b006b471be05e3 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 15:17:58.806133 20565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.209.254:37383, user_credentials={real_user=slave}} blocked reactor thread for 45171.1us
I20260926 15:17:58.814494 20657 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:17:58.816114 20657 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:17:58.846873 20680 catalog_manager.cc:1259] Loaded cluster ID: 35f9eac82b6a4e079b59634488351d3e
I20260926 15:17:58.846969 20680 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0cde5a183d734ade98b006b471be05e3: loading cluster ID for follower catalog manager: success
I20260926 15:17:58.850288 20680 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0cde5a183d734ade98b006b471be05e3: acquiring CA information for follower catalog manager: success
W20260926 15:17:58.851570 20680 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260926 15:17:58.852164 20657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd7f6988da5b4c66b14db69262673f07: Generated new TSK 0
I20260926 15:17:58.853042 20657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:17:59.511122 20653 catalog_manager.cc:1259] Loaded cluster ID: 35f9eac82b6a4e079b59634488351d3e
I20260926 15:17:59.511207 20653 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3005514e2a9e44a8ac3bfe4815a909ab: loading cluster ID for follower catalog manager: success
I20260926 15:17:59.513691 20653 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3005514e2a9e44a8ac3bfe4815a909ab: acquiring CA information for follower catalog manager: success
I20260926 15:17:59.515292 20653 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3005514e2a9e44a8ac3bfe4815a909ab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 15:17:59.853909 20680 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0cde5a183d734ade98b006b471be05e3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 15:18:00.028070 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 20399
I20260926 15:18:00.029178 20461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:18:00.130095 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 20399
I20260926 15:18:00.149818 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 20473
I20260926 15:18:00.151042 20535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:18:00.252115 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 20473
I20260926 15:18:00.271394 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 20547
I20260926 15:18:00.272480 20609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:18:00.371685 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 20547
2026-09-26T15:18:00Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4456 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:18:00 dist-test-slave-j6q7 krb5kdc[20693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:18:00 dist-test-slave-j6q7 krb5kdc[20693](info): set up 2 sockets
Sep 26 15:18:00 dist-test-slave-j6q7 krb5kdc[20693](info): commencing operation
krb5kdc: starting...
W20260926 15:18:01.365185 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:18:01 dist-test-slave-j6q7 krb5kdc[20693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:18:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:18:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:18:01.546502 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44299
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:44537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:44299
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:18:01.840791 20706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:18:01.841074 20706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:18:01.841220 20706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:18:01.848033 20706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:18:01.848119 20706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:18:01.848162 20706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:18:01.848217 20706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:18:01.848268 20706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:18:01.858143 20706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:44537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:44299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:18:01.859232 20706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:18:01.860430 20706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:18:01.872253 20711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:01.872225 20712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:01.872172 20714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:18:01.873273 20706 server_base.cc:1034] running on GCE node
I20260926 15:18:01.874112 20706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:18:01.875594 20706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:18:01.876863 20706 hybrid_clock.cc:648] HybridClock initialized: now 1790435881876778 us; error 53 us; skew 500 ppm
I20260926 15:18:01.877195 20706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:18:01 dist-test-slave-j6q7 krb5kdc[20693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435881, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:18:01.888655 20706 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:18:01.890045 20706 webserver.cc:460] Webserver started at http://127.6.209.254:46033/ using document root <none> and password file <none>
I20260926 15:18:01.890709 20706 fs_manager.cc:359] Metadata directory not provided
I20260926 15:18:01.890841 20706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:18:01.891172 20706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:18:01.894214 20706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "100e2fa9275f4eb8a20868305a7d0b35"
format_stamp: "Formatted at 2026-09-26 15:18:01 on dist-test-slave-j6q7"
server_key: "f37a50050612cddcc060476872f2ed9b"
server_key_iv: "7ed611d9507d24b8886fdb3473e593f9"
server_key_version: "encryptionkey@0"
I20260926 15:18:01.894948 20706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "100e2fa9275f4eb8a20868305a7d0b35"
format_stamp: "Formatted at 2026-09-26 15:18:01 on dist-test-slave-j6q7"
server_key: "f37a50050612cddcc060476872f2ed9b"
server_key_iv: "7ed611d9507d24b8886fdb3473e593f9"
server_key_version: "encryptionkey@0"
I20260926 15:18:01.899812 20706 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 15:18:01.902982 20721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:01.904183 20706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:18:01.904372 20706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "100e2fa9275f4eb8a20868305a7d0b35"
format_stamp: "Formatted at 2026-09-26 15:18:01 on dist-test-slave-j6q7"
server_key: "f37a50050612cddcc060476872f2ed9b"
server_key_iv: "7ed611d9507d24b8886fdb3473e593f9"
server_key_version: "encryptionkey@0"
I20260926 15:18:01.904562 20706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:18:01.914554 20706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:18:01.918085 20706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:18:01.918357 20706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:18:01.936116 20706 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:44299
I20260926 15:18:01.936137 20772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:44299 every 8 connection(s)
I20260926 15:18:01.937472 20706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:18:01.940955 20773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:18:01.943143 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 20706
I20260926 15:18:01.943389 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:18:01.944191 6983 external_mini_cluster.cc:1428] Setting key d9507a2f2c38e7f6ea4a6d4258d8c7b1
I20260926 15:18:01.953002 20773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 100e2fa9275f4eb8a20868305a7d0b35: Bootstrap starting.
I20260926 15:18:01.956648 20773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 100e2fa9275f4eb8a20868305a7d0b35: Neither blocks nor log segments found. Creating new log.
I20260926 15:18:01.957885 20773 log.cc:824] T 00000000000000000000000000000000 P 100e2fa9275f4eb8a20868305a7d0b35: Log is configured to *not* fsync() on all Append() calls
Sep 26 15:18:01 dist-test-slave-j6q7 krb5kdc[20693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:18:01.960644 20773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 100e2fa9275f4eb8a20868305a7d0b35: No bootstrap required, opened a new log
I20260926 15:18:01.966363 20776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:01.945938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58764 (local address 127.6.209.254:44299)
0926 15:18:01.946718 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:01.946729 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:01.946769 (+ 40us) server_negotiation.cc:407] Connection header received
0926 15:18:01.948366 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:01.948384 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:01.948719 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:01.949069 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:01.950756 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:01.951969 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:01.953855 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:01.954493 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:01.954621 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:01.959517 (+ 4896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:01.959524 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:01.959536 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:01.959573 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:01.962590 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:01.963287 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:01.963303 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:01.963308 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:01.963468 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:01.963856 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:01.963861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:01.963866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:01.964330 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:01.964637 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:01.965076 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 15:18:01.965468 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":359,"threads_started":1}
I20260926 15:18:01.968415 20773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 100e2fa9275f4eb8a20868305a7d0b35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100e2fa9275f4eb8a20868305a7d0b35" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44299 } }
I20260926 15:18:01.968794 20773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 100e2fa9275f4eb8a20868305a7d0b35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:18:01.968957 20773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 100e2fa9275f4eb8a20868305a7d0b35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100e2fa9275f4eb8a20868305a7d0b35, State: Initialized, Role: FOLLOWER
I20260926 15:18:01.969519 20773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 100e2fa9275f4eb8a20868305a7d0b35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100e2fa9275f4eb8a20868305a7d0b35" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44299 } }
I20260926 15:18:01.969777 20773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 100e2fa9275f4eb8a20868305a7d0b35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:18:01.969900 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:18:01.969911 20773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 100e2fa9275f4eb8a20868305a7d0b35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:18:01.970122 20773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 100e2fa9275f4eb8a20868305a7d0b35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:18:01.972189 20773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 100e2fa9275f4eb8a20868305a7d0b35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100e2fa9275f4eb8a20868305a7d0b35" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44299 } }
I20260926 15:18:01.972694 20773 leader_election.cc:302] T 00000000000000000000000000000000 P 100e2fa9275f4eb8a20868305a7d0b35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 100e2fa9275f4eb8a20868305a7d0b35; no voters:
I20260926 15:18:01.973343 20773 leader_election.cc:288] T 00000000000000000000000000000000 P 100e2fa9275f4eb8a20868305a7d0b35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:18:01.973443 20778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 100e2fa9275f4eb8a20868305a7d0b35 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:18:01.974038 20778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 100e2fa9275f4eb8a20868305a7d0b35 [term 1 LEADER]: Becoming Leader. State: Replica: 100e2fa9275f4eb8a20868305a7d0b35, State: Running, Role: LEADER
I20260926 15:18:01.974517 20778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 100e2fa9275f4eb8a20868305a7d0b35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100e2fa9275f4eb8a20868305a7d0b35" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44299 } }
I20260926 15:18:01.975420 20773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 100e2fa9275f4eb8a20868305a7d0b35 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:18:01.977075 20779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 100e2fa9275f4eb8a20868305a7d0b35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "100e2fa9275f4eb8a20868305a7d0b35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100e2fa9275f4eb8a20868305a7d0b35" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44299 } } }
I20260926 15:18:01.977420 20779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 100e2fa9275f4eb8a20868305a7d0b35 [sys.catalog]: This master's current role is: LEADER
I20260926 15:18:01.978073 20780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 100e2fa9275f4eb8a20868305a7d0b35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 100e2fa9275f4eb8a20868305a7d0b35. Latest consensus state: current_term: 1 leader_uuid: "100e2fa9275f4eb8a20868305a7d0b35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100e2fa9275f4eb8a20868305a7d0b35" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44299 } } }
I20260926 15:18:01.978375 20780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 100e2fa9275f4eb8a20868305a7d0b35 [sys.catalog]: This master's current role is: LEADER
I20260926 15:18:01.982120 20785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:18:01.986886 20785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:18:01.995059 20785 catalog_manager.cc:1347] Generated new cluster ID: 5fabff16dd7c41d297eaf7cd5ed8ace0
I20260926 15:18:01.995155 20785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:18:02.028170 20785 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:18:02.029405 20785 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:18:02.041097 20785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 100e2fa9275f4eb8a20868305a7d0b35: Generated new TSK 0
I20260926 15:18:02.041810 20785 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:18:02.081439 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 20706
I20260926 15:18:02.082553 20768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:18:02.175400 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 20706
2026-09-26T15:18:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1803 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:18:02 dist-test-slave-j6q7 krb5kdc[20801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:18:02 dist-test-slave-j6q7 krb5kdc[20801](info): set up 2 sockets
Sep 26 15:18:02 dist-test-slave-j6q7 krb5kdc[20801](info): commencing operation
krb5kdc: starting...
W20260926 15:18:03.197083 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:18:03 dist-test-slave-j6q7 krb5kdc[20801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:18:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:18:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:18:03.380882 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:40089
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:40089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:18:03.685793 20814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:18:03.686057 20814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:18:03.686252 20814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:18:03.693415 20814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:18:03.693538 20814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:18:03.693641 20814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:18:03.693746 20814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:18:03.693840 20814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:18:03.703284 20814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:40089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:40089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:18:03.704447 20814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:18:03.705706 20814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:18:03.717355 20819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:03.717402 20822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:03.717309 20820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:18:03.718084 20814 server_base.cc:1034] running on GCE node
I20260926 15:18:03.718856 20814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:18:03.720137 20814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:18:03.721371 20814 hybrid_clock.cc:648] HybridClock initialized: now 1790435883721311 us; error 44 us; skew 500 ppm
I20260926 15:18:03.721705 20814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:18:03 dist-test-slave-j6q7 krb5kdc[20801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435883, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:18:03.724728 20814 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:18:03.726142 20814 webserver.cc:460] Webserver started at http://127.6.209.254:34359/ using document root <none> and password file <none>
I20260926 15:18:03.726745 20814 fs_manager.cc:359] Metadata directory not provided
I20260926 15:18:03.726861 20814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:18:03.727109 20814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:18:03.730014 20814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "884244a56c274be9815711313673214a"
format_stamp: "Formatted at 2026-09-26 15:18:03 on dist-test-slave-j6q7"
server_key: "2d6a3b7f7f83efcdb789aed74a14d82c"
server_key_iv: "b211b9142906a90e5312f60a226e1347"
server_key_version: "encryptionkey@0"
I20260926 15:18:03.730775 20814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "884244a56c274be9815711313673214a"
format_stamp: "Formatted at 2026-09-26 15:18:03 on dist-test-slave-j6q7"
server_key: "2d6a3b7f7f83efcdb789aed74a14d82c"
server_key_iv: "b211b9142906a90e5312f60a226e1347"
server_key_version: "encryptionkey@0"
I20260926 15:18:03.735739 20814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 15:18:03.738984 20829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:03.740222 20814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 15:18:03.740365 20814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "884244a56c274be9815711313673214a"
format_stamp: "Formatted at 2026-09-26 15:18:03 on dist-test-slave-j6q7"
server_key: "2d6a3b7f7f83efcdb789aed74a14d82c"
server_key_iv: "b211b9142906a90e5312f60a226e1347"
server_key_version: "encryptionkey@0"
I20260926 15:18:03.740523 20814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:18:03.770941 20814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:18:03.774745 20814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:18:03.774989 20814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:18:03.793884 20814 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:40089
I20260926 15:18:03.793886 20880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:40089 every 8 connection(s)
I20260926 15:18:03.795178 20814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:18:03.796895 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 20814
I20260926 15:18:03.797103 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:18:03.797801 6983 external_mini_cluster.cc:1428] Setting key 0740115555a9c5e79da384fd603ef206
I20260926 15:18:03.798853 20881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:18:03.811894 20881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a: Bootstrap starting.
Sep 26 15:18:03 dist-test-slave-j6q7 krb5kdc[20801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:18:03.814710 20881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a: Neither blocks nor log segments found. Creating new log.
I20260926 15:18:03.815687 20881 log.cc:824] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a: Log is configured to *not* fsync() on all Append() calls
I20260926 15:18:03.818063 20881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a: No bootstrap required, opened a new log
I20260926 15:18:03.820034 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:03.799738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37262 (local address 127.6.209.254:40089)
0926 15:18:03.800589 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:03.800601 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:03.800640 (+ 39us) server_negotiation.cc:407] Connection header received
0926 15:18:03.803062 (+ 2422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:03.803083 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:03.803452 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:03.803765 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:03.805424 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:03.806758 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:03.808441 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:03.809187 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:03.809331 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:03.813692 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:03.813698 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:03.813709 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:03.813741 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:03.816599 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:03.817247 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:03.817257 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:03.817262 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:03.817407 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:03.817759 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:03.817763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:03.817765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:03.818218 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:03.818524 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:03.818858 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 15:18:03.819198 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":391,"threads_started":1}
I20260926 15:18:03.825591 20881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884244a56c274be9815711313673214a" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40089 } }
I20260926 15:18:03.825937 20881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:18:03.826043 20881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 884244a56c274be9815711313673214a, State: Initialized, Role: FOLLOWER
I20260926 15:18:03.826507 20881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884244a56c274be9815711313673214a" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40089 } }
I20260926 15:18:03.826730 20881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:18:03.826869 20881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:18:03.827044 20881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:18:03.829084 20881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884244a56c274be9815711313673214a" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40089 } }
I20260926 15:18:03.829488 20881 leader_election.cc:302] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 884244a56c274be9815711313673214a; no voters:
I20260926 15:18:03.829985 20881 leader_election.cc:288] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:18:03.830089 20886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:18:03.830689 20886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 1 LEADER]: Becoming Leader. State: Replica: 884244a56c274be9815711313673214a, State: Running, Role: LEADER
I20260926 15:18:03.831182 20886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884244a56c274be9815711313673214a" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40089 } }
I20260926 15:18:03.831784 20881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:18:03.833770 20888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 884244a56c274be9815711313673214a. Latest consensus state: current_term: 1 leader_uuid: "884244a56c274be9815711313673214a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884244a56c274be9815711313673214a" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40089 } } }
I20260926 15:18:03.834197 20888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [sys.catalog]: This master's current role is: LEADER
I20260926 15:18:03.833882 20887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "884244a56c274be9815711313673214a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884244a56c274be9815711313673214a" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40089 } } }
I20260926 15:18:03.834514 20887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [sys.catalog]: This master's current role is: LEADER
I20260926 15:18:03.835955 20895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:18:03.839862 20895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:18:03.847478 20895 catalog_manager.cc:1347] Generated new cluster ID: aeea6295143a4ae388e994bdb41048d1
I20260926 15:18:03.847581 20895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:18:03.855118 20895 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:18:03.856245 20895 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:18:03.861514 20895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a: Generated new TSK 0
I20260926 15:18:03.862103 20895 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
I20260926 15:18:03.912583 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:40089
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:18:04.209118 20907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:18:04.209385 20907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:18:04.209543 20907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:18:04.215785 20907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:18:04.215904 20907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:18:04.216068 20907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:18:04.224786 20907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:40089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:18:04.225978 20907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:18:04.227252 20907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:18:04.239557 20913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:04.239593 20915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:04.239650 20912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:18:04.240548 20907 server_base.cc:1034] running on GCE node
I20260926 15:18:04.241103 20907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:18:04.241885 20907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:18:04.243072 20907 hybrid_clock.cc:648] HybridClock initialized: now 1790435884243040 us; error 29 us; skew 500 ppm
I20260926 15:18:04.243355 20907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:18:04 dist-test-slave-j6q7 krb5kdc[20801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435884, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:18:04.246111 20907 init.cc:377] Logged in from keytab as kudu/127.6.209.193@KRBTEST.COM (short username kudu)
I20260926 15:18:04.247308 20907 webserver.cc:460] Webserver started at http://127.6.209.193:38465/ using document root <none> and password file <none>
I20260926 15:18:04.247918 20907 fs_manager.cc:359] Metadata directory not provided
I20260926 15:18:04.248034 20907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:18:04.248246 20907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:18:04.250948 20907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "5161fb9c84864bd4bf98f2ce2e724bfa"
format_stamp: "Formatted at 2026-09-26 15:18:04 on dist-test-slave-j6q7"
server_key: "79d81f1983a94a17bc9f9518b67a8f55"
server_key_iv: "66bef5cd196840de3154e2e823da9791"
server_key_version: "encryptionkey@0"
I20260926 15:18:04.251637 20907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "5161fb9c84864bd4bf98f2ce2e724bfa"
format_stamp: "Formatted at 2026-09-26 15:18:04 on dist-test-slave-j6q7"
server_key: "79d81f1983a94a17bc9f9518b67a8f55"
server_key_iv: "66bef5cd196840de3154e2e823da9791"
server_key_version: "encryptionkey@0"
I20260926 15:18:04.255987 20907 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 15:18:04.259002 20922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:04.260164 20907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 15:18:04.260315 20907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "5161fb9c84864bd4bf98f2ce2e724bfa"
format_stamp: "Formatted at 2026-09-26 15:18:04 on dist-test-slave-j6q7"
server_key: "79d81f1983a94a17bc9f9518b67a8f55"
server_key_iv: "66bef5cd196840de3154e2e823da9791"
server_key_version: "encryptionkey@0"
I20260926 15:18:04.260500 20907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:18:04.272539 20907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:18:04.275947 20907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:18:04.276191 20907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:18:04.277000 20907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:18:04.278124 20907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:18:04.278188 20907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:04.278331 20907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:18:04.278403 20907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:04.308425 20907 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:43579
I20260926 15:18:04.308458 21034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:43579 every 8 connection(s)
I20260926 15:18:04.309518 20907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:18:04.311278 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 20907
I20260926 15:18:04.311468 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:18:04.312207 6983 external_mini_cluster.cc:1428] Setting key 53f23533a983603d96b5bf329c50a57f
Sep 26 15:18:04 dist-test-slave-j6q7 krb5kdc[20801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435884, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:18:04.328615 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:04.311903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:59781 (local address 127.6.209.254:40089)
0926 15:18:04.312103 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:04.312108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:04.313367 (+ 1259us) server_negotiation.cc:407] Connection header received
0926 15:18:04.314939 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:04.314945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:04.315028 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:04.315128 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:04.316393 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:04.317405 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:04.318697 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:04.319331 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:04.319444 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:04.323325 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:04.323330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:04.323335 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:04.323370 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:04.325715 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:04.326407 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:04.326412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:04.326415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:04.326552 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:04.327028 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:04.327032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:04.327033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:04.327284 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:04.327441 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:04.328044 (+ 603us) server_negotiation.cc:299] Negotiation successful
0926 15:18:04.328264 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 15:18:04.329494 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:04.312487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.193:59781)
0926 15:18:04.313223 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:04.313265 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:04.314708 (+ 1443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:04.315288 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:04.315295 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:04.315749 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:18:04.316246 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:04.316260 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:04.317544 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:04.317547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:04.318570 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:04.318581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:04.319533 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:04.319541 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:04.319784 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:04.320573 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:18:04.320605 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:18:04.323113 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:18:04.325865 (+ 2752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:18:04.325872 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:18:04.325884 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:18:04.326291 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:18:04.326680 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:18:04.326685 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:18:04.326689 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:18:04.326890 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:18:04.327416 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:18:04.327422 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:18:04.327718 (+ 296us) client_negotiation.cc:769] Sending connection context
0926 15:18:04.327954 (+ 236us) client_negotiation.cc:240] Negotiation successful
0926 15:18:04.328447 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":326,"threads_started":1}
I20260926 15:18:04.330891 21035 heartbeater.cc:347] Connected to a master server at 127.6.209.254:40089
I20260926 15:18:04.331199 21035 heartbeater.cc:464] Registering TS with master...
I20260926 15:18:04.332082 21035 heartbeater.cc:511] Master 127.6.209.254:40089 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.209.194@KRBTEST.COM; defaulting to no policy
I20260926 15:18:04.334259 20846 ts_manager.cc:194] Registered new tserver with Master: 5161fb9c84864bd4bf98f2ce2e724bfa (127.6.209.193:43579)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.194@KRBTEST.COM" created.
I20260926 15:18:04.335988 20846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.193@KRBTEST.COM'} at 127.6.209.193:59781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
I20260926 15:18:04.357975 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:40089
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:18:04.655766 21040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:18:04.655995 21040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:18:04.656076 21040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:18:04.662511 21040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:18:04.662599 21040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:18:04.662703 21040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:18:04.671008 21040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:40089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:18:04.672019 21040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:18:04.673192 21040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:18:04.685412 21045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:04.685456 21046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:04.685438 21048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:18:04.686067 21040 server_base.cc:1034] running on GCE node
I20260926 15:18:04.686651 21040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:18:04.687397 21040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:18:04.688598 21040 hybrid_clock.cc:648] HybridClock initialized: now 1790435884688560 us; error 45 us; skew 500 ppm
I20260926 15:18:04.688925 21040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:18:04 dist-test-slave-j6q7 krb5kdc[20801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435884, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:18:04.691582 21040 init.cc:377] Logged in from keytab as kudu/127.6.209.194@KRBTEST.COM (short username kudu)
I20260926 15:18:04.692926 21040 webserver.cc:460] Webserver started at http://127.6.209.194:33857/ using document root <none> and password file <none>
I20260926 15:18:04.693490 21040 fs_manager.cc:359] Metadata directory not provided
I20260926 15:18:04.693608 21040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:18:04.693861 21040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:18:04.696149 21040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe"
format_stamp: "Formatted at 2026-09-26 15:18:04 on dist-test-slave-j6q7"
server_key: "7a8dc3e8c6ae79ba4032bc55343ce88b"
server_key_iv: "b5fc3a54d08c31557fa0a1bda370fd88"
server_key_version: "encryptionkey@0"
I20260926 15:18:04.696787 21040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe"
format_stamp: "Formatted at 2026-09-26 15:18:04 on dist-test-slave-j6q7"
server_key: "7a8dc3e8c6ae79ba4032bc55343ce88b"
server_key_iv: "b5fc3a54d08c31557fa0a1bda370fd88"
server_key_version: "encryptionkey@0"
I20260926 15:18:04.701004 21040 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:18:04.703980 21055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:04.705087 21040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260926 15:18:04.705236 21040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe"
format_stamp: "Formatted at 2026-09-26 15:18:04 on dist-test-slave-j6q7"
server_key: "7a8dc3e8c6ae79ba4032bc55343ce88b"
server_key_iv: "b5fc3a54d08c31557fa0a1bda370fd88"
server_key_version: "encryptionkey@0"
I20260926 15:18:04.705397 21040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:18:04.725780 21040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:18:04.728971 21040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:18:04.729203 21040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:18:04.729936 21040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:18:04.731024 21040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:18:04.731112 21040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:04.731233 21040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:18:04.731330 21040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:04.760632 21040 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:34839
I20260926 15:18:04.760702 21167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:34839 every 8 connection(s)
I20260926 15:18:04.761737 21040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:18:04.764293 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 21040
I20260926 15:18:04.764472 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:18:04.764940 6983 external_mini_cluster.cc:1428] Setting key 50a7e9c2ec8453906a18967f1e16c2a1
Sep 26 15:18:04 dist-test-slave-j6q7 krb5kdc[20801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435884, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:18:04.780956 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:04.764073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:42129 (local address 127.6.209.254:40089)
0926 15:18:04.764303 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:04.764310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:04.765428 (+ 1118us) server_negotiation.cc:407] Connection header received
0926 15:18:04.767169 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:04.767174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:04.767245 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:04.767372 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:04.768560 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:04.769573 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:04.771427 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:04.772162 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:04.772289 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:04.775719 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:04.775725 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:04.775730 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:04.775790 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:04.778154 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:04.778895 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:04.778900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:04.778903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:04.779011 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:04.779373 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:04.779380 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:04.779384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:04.779610 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:04.779758 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:04.780356 (+ 598us) server_negotiation.cc:299] Negotiation successful
0926 15:18:04.780569 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 15:18:04.781916 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:04.764602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.194:42129)
0926 15:18:04.765277 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:04.765315 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:04.766944 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:04.767526 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:04.767533 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:04.767955 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:18:04.768413 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:04.768427 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:04.769705 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:04.769712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:04.771276 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:04.771291 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:04.772365 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:04.772373 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:04.772645 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:04.773364 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:18:04.773397 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:18:04.775523 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:18:04.778292 (+ 2769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:18:04.778299 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:18:04.778311 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:18:04.778786 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:18:04.779106 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:18:04.779111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:18:04.779114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:18:04.779269 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:18:04.779733 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:18:04.779740 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:18:04.780006 (+ 266us) client_negotiation.cc:769] Sending connection context
0926 15:18:04.780274 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 15:18:04.780728 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":313,"threads_started":1}
I20260926 15:18:04.783291 21168 heartbeater.cc:347] Connected to a master server at 127.6.209.254:40089
I20260926 15:18:04.783541 21168 heartbeater.cc:464] Registering TS with master...
I20260926 15:18:04.784149 21168 heartbeater.cc:511] Master 127.6.209.254:40089 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.209.195@KRBTEST.COM; defaulting to no policy
I20260926 15:18:04.785450 20846 ts_manager.cc:194] Registered new tserver with Master: 6c590b9a8f8c4be7a0dee9d3f30eb8fe (127.6.209.194:34839)
I20260926 15:18:04.786201 20846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.194@KRBTEST.COM'} at 127.6.209.194:42129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
I20260926 15:18:04.810058 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:40089
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:18:05.100446 21173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:18:05.100678 21173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:18:05.100764 21173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:18:05.107718 21173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:18:05.107836 21173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:18:05.108006 21173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:18:05.116712 21173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:40089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:18:05.117691 21173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:18:05.118810 21173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:18:05.130358 21178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:05.130391 21179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:05.130334 21181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:18:05.131028 21173 server_base.cc:1034] running on GCE node
I20260926 15:18:05.131685 21173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:18:05.132527 21173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:18:05.133715 21173 hybrid_clock.cc:648] HybridClock initialized: now 1790435885133676 us; error 36 us; skew 500 ppm
I20260926 15:18:05.133976 21173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:18:05 dist-test-slave-j6q7 krb5kdc[20801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435885, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:18:05.136667 21173 init.cc:377] Logged in from keytab as kudu/127.6.209.195@KRBTEST.COM (short username kudu)
I20260926 15:18:05.137857 21173 webserver.cc:460] Webserver started at http://127.6.209.195:36133/ using document root <none> and password file <none>
I20260926 15:18:05.138417 21173 fs_manager.cc:359] Metadata directory not provided
I20260926 15:18:05.138530 21173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:18:05.138777 21173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:18:05.141305 21173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "c9ea8357080a4741a2d9e555265ccbe9"
format_stamp: "Formatted at 2026-09-26 15:18:05 on dist-test-slave-j6q7"
server_key: "05960ef35ec7dfe6e78a3e2bf9a5c885"
server_key_iv: "d41a0ce4f35f8a07ee2a75186c4739b2"
server_key_version: "encryptionkey@0"
I20260926 15:18:05.142012 21173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "c9ea8357080a4741a2d9e555265ccbe9"
format_stamp: "Formatted at 2026-09-26 15:18:05 on dist-test-slave-j6q7"
server_key: "05960ef35ec7dfe6e78a3e2bf9a5c885"
server_key_iv: "d41a0ce4f35f8a07ee2a75186c4739b2"
server_key_version: "encryptionkey@0"
I20260926 15:18:05.146575 21173 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 15:18:05.149386 21188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:05.150440 21173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:18:05.150583 21173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "c9ea8357080a4741a2d9e555265ccbe9"
format_stamp: "Formatted at 2026-09-26 15:18:05 on dist-test-slave-j6q7"
server_key: "05960ef35ec7dfe6e78a3e2bf9a5c885"
server_key_iv: "d41a0ce4f35f8a07ee2a75186c4739b2"
server_key_version: "encryptionkey@0"
I20260926 15:18:05.150749 21173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:18:05.168285 21173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:18:05.171737 21173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:18:05.172014 21173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:18:05.172883 21173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:18:05.173926 21173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:18:05.173997 21173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:05.174155 21173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:18:05.174198 21173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:05.202629 21173 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:36831
I20260926 15:18:05.202698 21300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:36831 every 8 connection(s)
I20260926 15:18:05.203696 21173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:18:05.207561 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 21173
I20260926 15:18:05.207732 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:18:05.208534 6983 external_mini_cluster.cc:1428] Setting key 2fbc24d974edf5cccda01401d38fe2af
Sep 26 15:18:05 dist-test-slave-j6q7 krb5kdc[20801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435885, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:18:05.222815 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:05.205773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:45397 (local address 127.6.209.254:40089)
0926 15:18:05.205957 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:05.205963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:05.206883 (+ 920us) server_negotiation.cc:407] Connection header received
0926 15:18:05.209096 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:05.209103 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:05.209188 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:05.209314 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:05.211287 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.212360 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.213774 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.214422 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.214565 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:05.217794 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:05.217798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:05.217801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:05.217829 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:05.220190 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:05.220755 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:05.220758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:05.220760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:05.220930 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:05.221265 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:05.221268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:05.221270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:05.221533 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:05.221640 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:05.222251 (+ 611us) server_negotiation.cc:299] Negotiation successful
0926 15:18:05.222546 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 15:18:05.223564 21303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:05.206179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.195:45397)
0926 15:18:05.206744 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:05.206779 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:05.208780 (+ 2001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:05.209710 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:05.209723 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:05.210377 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:18:05.211073 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:05.211098 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.212537 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.212540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:05.213622 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:05.213632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.214632 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.214639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:05.214819 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:05.215392 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:18:05.215415 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:18:05.217615 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:18:05.220293 (+ 2678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:18:05.220298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:18:05.220308 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:18:05.220655 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:18:05.221020 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:18:05.221023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:18:05.221025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:18:05.221172 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:18:05.221631 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:18:05.221636 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:18:05.221885 (+ 249us) client_negotiation.cc:769] Sending connection context
0926 15:18:05.222150 (+ 265us) client_negotiation.cc:240] Negotiation successful
0926 15:18:05.222590 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":225,"threads_started":1}
I20260926 15:18:05.224615 21301 heartbeater.cc:347] Connected to a master server at 127.6.209.254:40089
I20260926 15:18:05.224923 21301 heartbeater.cc:464] Registering TS with master...
I20260926 15:18:05.225524 21301 heartbeater.cc:511] Master 127.6.209.254:40089 requested a full tablet report, sending...
I20260926 15:18:05.226792 20846 ts_manager.cc:194] Registered new tserver with Master: c9ea8357080a4741a2d9e555265ccbe9 (127.6.209.195:36831)
I20260926 15:18:05.227506 20846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.195@KRBTEST.COM'} at 127.6.209.195:45397
I20260926 15:18:05.235497 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:18:05.250001 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:05.239265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37268 (local address 127.6.209.254:40089)
0926 15:18:05.239490 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:05.239496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:05.239757 (+ 261us) server_negotiation.cc:407] Connection header received
0926 15:18:05.239863 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:05.239866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:05.239916 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:05.239987 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:05.241314 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.241986 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.243173 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.243601 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.243683 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:05.245870 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:05.245874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:05.245877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:05.245902 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:05.247935 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:05.248441 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:05.248444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:05.248446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:05.248554 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:05.248845 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:05.248851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:05.248853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:05.249105 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:05.249278 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:05.249529 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 15:18:05.249685 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 15:18:05.253487 20846 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:18:05.259481 20846 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:18:05.281533 21313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:05.269725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:36831 (local address 127.0.0.1:35936)
0926 15:18:05.270362 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:05.270404 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:05.271238 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:05.272187 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:05.272197 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:05.272236 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:18:05.272750 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:05.272764 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.275232 (+ 2468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.275239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:05.278129 (+ 2890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:05.278147 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.280340 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.280348 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:05.280549 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:05.280621 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 15:18:05.280740 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 15:18:05.280998 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":296,"threads_started":1}
I20260926 15:18:05.285607 21314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:05.270045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35936 (local address 127.6.209.195:36831)
0926 15:18:05.270869 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:05.270878 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:05.270920 (+ 42us) server_negotiation.cc:407] Connection header received
0926 15:18:05.271613 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:05.271623 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:05.271917 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:05.272994 (+ 1077us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:18:05.273310 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.275064 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.278353 (+ 3289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.280107 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.284599 (+ 4492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:05.284913 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:05.285000 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 15:18:05.285187 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":550,"threads_started":1}
I20260926 15:18:05.289287 21313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:05.270333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:34839 (local address 127.0.0.1:39576)
0926 15:18:05.281803 (+ 11470us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:05.281824 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:05.281945 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:05.282985 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:05.282992 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:05.283022 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:18:05.283228 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:05.283239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.285084 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.285089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:05.286986 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:05.287001 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.288546 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.288551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:05.288757 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:05.288785 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 15:18:05.288891 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 15:18:05.289009 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11348,"mutex_wait_us":29,"thread_start_us":13158,"threads_started":1}
I20260926 15:18:05.289319 21315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:05.270769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39576 (local address 127.6.209.194:34839)
0926 15:18:05.271235 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:05.271242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:05.282309 (+ 11067us) server_negotiation.cc:407] Connection header received
0926 15:18:05.282435 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:05.282486 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:05.282765 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:05.283400 (+ 635us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:18:05.283610 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.284942 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.287170 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.288389 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.288515 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:05.288662 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:05.288929 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 15:18:05.289063 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":199,"threads_started":1}
I20260926 15:18:05.289250 21236 tablet_service.cc:1467] Processing CreateTablet for tablet ba83d188a06f40ec9d26d4429031fb63 (DEFAULT_TABLE table=test-table [id=751f90f4c13842ae96a417216fd480e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:18:05.290544 21236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba83d188a06f40ec9d26d4429031fb63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:18:05.291963 21316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:05.271474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:43579 (local address 127.0.0.1:45744)
0926 15:18:05.283743 (+ 12269us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:05.283758 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:05.283858 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:05.285044 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:05.285049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:05.285074 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:18:05.285463 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:05.285473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.287170 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.287174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:05.289650 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:05.289662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.291170 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.291176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:05.291353 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:05.291383 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 15:18:05.291474 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 15:18:05.291590 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12146,"mutex_wait_us":1,"thread_start_us":14706,"threads_started":1}
I20260926 15:18:05.292069 21103 tablet_service.cc:1467] Processing CreateTablet for tablet ba83d188a06f40ec9d26d4429031fb63 (DEFAULT_TABLE table=test-table [id=751f90f4c13842ae96a417216fd480e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:18:05.292738 21317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:05.271373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45744 (local address 127.6.209.193:43579)
0926 15:18:05.284254 (+ 12881us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:05.284261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:05.284282 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:18:05.284358 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:05.284364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:05.284547 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:05.284691 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:18:05.285650 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.287032 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.289788 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.290983 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.291106 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:05.291246 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:05.292266 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0926 15:18:05.292418 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12743,"thread_start_us":13467,"threads_started":1}
I20260926 15:18:05.293278 21103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba83d188a06f40ec9d26d4429031fb63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:18:05.295807 20970 tablet_service.cc:1467] Processing CreateTablet for tablet ba83d188a06f40ec9d26d4429031fb63 (DEFAULT_TABLE table=test-table [id=751f90f4c13842ae96a417216fd480e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:18:05.296895 20970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba83d188a06f40ec9d26d4429031fb63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:18:05.301923 21319 tablet_bootstrap.cc:492] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9: Bootstrap starting.
I20260926 15:18:05.304744 21319 tablet_bootstrap.cc:654] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9: Neither blocks nor log segments found. Creating new log.
I20260926 15:18:05.305855 21319 log.cc:824] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9: Log is configured to *not* fsync() on all Append() calls
I20260926 15:18:05.306006 21320 tablet_bootstrap.cc:492] T ba83d188a06f40ec9d26d4429031fb63 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe: Bootstrap starting.
I20260926 15:18:05.309170 21320 tablet_bootstrap.cc:654] T ba83d188a06f40ec9d26d4429031fb63 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe: Neither blocks nor log segments found. Creating new log.
I20260926 15:18:05.309707 21319 tablet_bootstrap.cc:492] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9: No bootstrap required, opened a new log
I20260926 15:18:05.310009 21319 ts_tablet_manager.cc:1397] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 15:18:05.310405 21320 log.cc:824] T ba83d188a06f40ec9d26d4429031fb63 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe: Log is configured to *not* fsync() on all Append() calls
I20260926 15:18:05.319265 21320 tablet_bootstrap.cc:492] T ba83d188a06f40ec9d26d4429031fb63 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe: No bootstrap required, opened a new log
I20260926 15:18:05.319543 21320 ts_tablet_manager.cc:1397] T ba83d188a06f40ec9d26d4429031fb63 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20260926 15:18:05.321451 21319 raft_consensus.cc:348] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ea8357080a4741a2d9e555265ccbe9" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36831 } } peers { permanent_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43579 } } peers { permanent_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34839 } }
I20260926 15:18:05.322466 21322 tablet_bootstrap.cc:492] T ba83d188a06f40ec9d26d4429031fb63 P 5161fb9c84864bd4bf98f2ce2e724bfa: Bootstrap starting.
I20260926 15:18:05.325790 21322 tablet_bootstrap.cc:654] T ba83d188a06f40ec9d26d4429031fb63 P 5161fb9c84864bd4bf98f2ce2e724bfa: Neither blocks nor log segments found. Creating new log.
I20260926 15:18:05.326777 21322 log.cc:824] T ba83d188a06f40ec9d26d4429031fb63 P 5161fb9c84864bd4bf98f2ce2e724bfa: Log is configured to *not* fsync() on all Append() calls
I20260926 15:18:05.327024 21319 raft_consensus.cc:374] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:18:05.327122 21319 raft_consensus.cc:729] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ea8357080a4741a2d9e555265ccbe9, State: Initialized, Role: FOLLOWER
I20260926 15:18:05.327723 21319 consensus_queue.cc:260] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ea8357080a4741a2d9e555265ccbe9" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36831 } } peers { permanent_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43579 } } peers { permanent_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34839 } }
I20260926 15:18:05.328634 21301 heartbeater.cc:503] Master 127.6.209.254:40089 was elected leader, sending a full tablet report...
I20260926 15:18:05.328709 21320 raft_consensus.cc:348] T ba83d188a06f40ec9d26d4429031fb63 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ea8357080a4741a2d9e555265ccbe9" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36831 } } peers { permanent_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43579 } } peers { permanent_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34839 } }
I20260926 15:18:05.329072 21320 raft_consensus.cc:374] T ba83d188a06f40ec9d26d4429031fb63 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:18:05.329149 21320 raft_consensus.cc:729] T ba83d188a06f40ec9d26d4429031fb63 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c590b9a8f8c4be7a0dee9d3f30eb8fe, State: Initialized, Role: FOLLOWER
I20260926 15:18:05.329245 21319 ts_tablet_manager.cc:1428] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9: Time spent starting tablet: real 0.019s user 0.013s sys 0.001s
I20260926 15:18:05.329324 21322 tablet_bootstrap.cc:492] T ba83d188a06f40ec9d26d4429031fb63 P 5161fb9c84864bd4bf98f2ce2e724bfa: No bootstrap required, opened a new log
I20260926 15:18:05.329527 21322 ts_tablet_manager.cc:1397] T ba83d188a06f40ec9d26d4429031fb63 P 5161fb9c84864bd4bf98f2ce2e724bfa: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260926 15:18:05.329717 21320 consensus_queue.cc:260] T ba83d188a06f40ec9d26d4429031fb63 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ea8357080a4741a2d9e555265ccbe9" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36831 } } peers { permanent_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43579 } } peers { permanent_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34839 } }
I20260926 15:18:05.330621 21168 heartbeater.cc:503] Master 127.6.209.254:40089 was elected leader, sending a full tablet report...
I20260926 15:18:05.332249 21320 ts_tablet_manager.cc:1428] T ba83d188a06f40ec9d26d4429031fb63 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe: Time spent starting tablet: real 0.013s user 0.009s sys 0.001s
I20260926 15:18:05.337499 21322 raft_consensus.cc:348] T ba83d188a06f40ec9d26d4429031fb63 P 5161fb9c84864bd4bf98f2ce2e724bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ea8357080a4741a2d9e555265ccbe9" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36831 } } peers { permanent_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43579 } } peers { permanent_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34839 } }
I20260926 15:18:05.337778 21322 raft_consensus.cc:374] T ba83d188a06f40ec9d26d4429031fb63 P 5161fb9c84864bd4bf98f2ce2e724bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:18:05.337865 21322 raft_consensus.cc:729] T ba83d188a06f40ec9d26d4429031fb63 P 5161fb9c84864bd4bf98f2ce2e724bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5161fb9c84864bd4bf98f2ce2e724bfa, State: Initialized, Role: FOLLOWER
I20260926 15:18:05.338274 21322 consensus_queue.cc:260] T ba83d188a06f40ec9d26d4429031fb63 P 5161fb9c84864bd4bf98f2ce2e724bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ea8357080a4741a2d9e555265ccbe9" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36831 } } peers { permanent_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43579 } } peers { permanent_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34839 } }
I20260926 15:18:05.338472 21035 heartbeater.cc:503] Master 127.6.209.254:40089 was elected leader, sending a full tablet report...
I20260926 15:18:05.339619 21322 ts_tablet_manager.cc:1428] T ba83d188a06f40ec9d26d4429031fb63 P 5161fb9c84864bd4bf98f2ce2e724bfa: Time spent starting tablet: real 0.010s user 0.008s sys 0.001s
W20260926 15:18:05.455112 21302 tablet.cc:2369] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:18:05.467270 21325 raft_consensus.cc:482] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:18:05.467537 21325 raft_consensus.cc:504] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ea8357080a4741a2d9e555265ccbe9" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36831 } } peers { permanent_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43579 } } peers { permanent_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34839 } }
I20260926 15:18:05.469080 21325 leader_election.cc:288] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5161fb9c84864bd4bf98f2ce2e724bfa (127.6.209.193:43579), 6c590b9a8f8c4be7a0dee9d3f30eb8fe (127.6.209.194:34839)
I20260926 15:18:05.474520 21315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:05.469666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:47729 (local address 127.6.209.194:34839)
0926 15:18:05.469843 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:05.469846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:05.469880 (+ 34us) server_negotiation.cc:407] Connection header received
0926 15:18:05.469943 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:05.469948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:05.470026 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:05.470111 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:18:05.470659 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.471271 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.472862 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.473661 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.473819 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:05.473893 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:05.474163 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 15:18:05.474277 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 15:18:05.474524 21303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:05.469298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:34839 (local address 127.6.209.195:47729)
0926 15:18:05.469452 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:05.469470 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:05.469611 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:05.470263 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:05.470268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:05.470300 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:18:05.470496 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:05.470509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.471432 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.471440 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:05.472689 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:05.472700 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.473807 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.473812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:05.474002 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:05.474024 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 15:18:05.474138 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 15:18:05.474260 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 15:18:05.475328 21328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:05.469302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:43579 (local address 127.6.209.195:48799)
0926 15:18:05.470333 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:05.470351 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:05.470542 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:05.470837 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:05.470841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:05.470861 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:18:05.471138 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:05.471145 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.472132 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.472138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:05.473541 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:05.473552 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.474649 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.474655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:05.474831 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:05.474870 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 15:18:05.474958 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 15:18:05.475074 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"mutex_wait_us":391,"thread_start_us":310,"threads_started":1}
I20260926 15:18:05.475359 21317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:05.469410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:48799 (local address 127.6.209.193:43579)
0926 15:18:05.469624 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:05.469631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:05.470420 (+ 789us) server_negotiation.cc:407] Connection header received
0926 15:18:05.470686 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:05.470690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:05.470742 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:05.470827 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:18:05.471285 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.471993 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.473707 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.474459 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.474636 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:05.474700 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:05.474992 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 15:18:05.475114 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 15:18:05.475600 21123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba83d188a06f40ec9d26d4429031fb63" candidate_uuid: "c9ea8357080a4741a2d9e555265ccbe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" is_pre_election: true
I20260926 15:18:05.476152 21123 raft_consensus.cc:2454] T ba83d188a06f40ec9d26d4429031fb63 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9ea8357080a4741a2d9e555265ccbe9 in term 0.
I20260926 15:18:05.476346 20990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba83d188a06f40ec9d26d4429031fb63" candidate_uuid: "c9ea8357080a4741a2d9e555265ccbe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" is_pre_election: true
I20260926 15:18:05.476846 21191 leader_election.cc:302] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c590b9a8f8c4be7a0dee9d3f30eb8fe, c9ea8357080a4741a2d9e555265ccbe9; no voters:
I20260926 15:18:05.476929 20990 raft_consensus.cc:2454] T ba83d188a06f40ec9d26d4429031fb63 P 5161fb9c84864bd4bf98f2ce2e724bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9ea8357080a4741a2d9e555265ccbe9 in term 0.
I20260926 15:18:05.477264 21325 raft_consensus.cc:2781] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:18:05.477435 21325 raft_consensus.cc:482] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:18:05.477571 21325 raft_consensus.cc:3037] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:18:05.479661 21325 raft_consensus.cc:504] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ea8357080a4741a2d9e555265ccbe9" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36831 } } peers { permanent_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43579 } } peers { permanent_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34839 } }
I20260926 15:18:05.480206 21325 leader_election.cc:288] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9 [CANDIDATE]: Term 1 election: Requested vote from peers 5161fb9c84864bd4bf98f2ce2e724bfa (127.6.209.193:43579), 6c590b9a8f8c4be7a0dee9d3f30eb8fe (127.6.209.194:34839)
I20260926 15:18:05.480610 20990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba83d188a06f40ec9d26d4429031fb63" candidate_uuid: "c9ea8357080a4741a2d9e555265ccbe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa"
I20260926 15:18:05.480710 21123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba83d188a06f40ec9d26d4429031fb63" candidate_uuid: "c9ea8357080a4741a2d9e555265ccbe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe"
I20260926 15:18:05.480782 20990 raft_consensus.cc:3037] T ba83d188a06f40ec9d26d4429031fb63 P 5161fb9c84864bd4bf98f2ce2e724bfa [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:18:05.480918 21123 raft_consensus.cc:3037] T ba83d188a06f40ec9d26d4429031fb63 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:18:05.482797 20990 raft_consensus.cc:2454] T ba83d188a06f40ec9d26d4429031fb63 P 5161fb9c84864bd4bf98f2ce2e724bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9ea8357080a4741a2d9e555265ccbe9 in term 1.
I20260926 15:18:05.483019 21123 raft_consensus.cc:2454] T ba83d188a06f40ec9d26d4429031fb63 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9ea8357080a4741a2d9e555265ccbe9 in term 1.
I20260926 15:18:05.483177 21190 leader_election.cc:302] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5161fb9c84864bd4bf98f2ce2e724bfa, c9ea8357080a4741a2d9e555265ccbe9; no voters:
I20260926 15:18:05.483486 21325 raft_consensus.cc:2781] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:18:05.483914 21325 raft_consensus.cc:686] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9 [term 1 LEADER]: Becoming Leader. State: Replica: c9ea8357080a4741a2d9e555265ccbe9, State: Running, Role: LEADER
I20260926 15:18:05.484337 21325 consensus_queue.cc:237] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ea8357080a4741a2d9e555265ccbe9" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36831 } } peers { permanent_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43579 } } peers { permanent_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34839 } }
I20260926 15:18:05.488416 20845 catalog_manager.cc:5495] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9ea8357080a4741a2d9e555265ccbe9 (127.6.209.195). New cstate: current_term: 1 leader_uuid: "c9ea8357080a4741a2d9e555265ccbe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ea8357080a4741a2d9e555265ccbe9" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34839 } health_report { overall_health: UNKNOWN } } }
W20260926 15:18:05.513474 21169 tablet.cc:2369] T ba83d188a06f40ec9d26d4429031fb63 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:18:05.526168 21314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:05.521598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35942 (local address 127.6.209.195:36831)
0926 15:18:05.521760 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:05.521764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:05.521849 (+ 85us) server_negotiation.cc:407] Connection header received
0926 15:18:05.522008 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:05.522012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:05.522089 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:05.522204 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:18:05.522537 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.523167 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.524395 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.524734 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.524851 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:05.525224 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:18:05.525757 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:18:05.525860 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:05.525928 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 15:18:05.526004 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 15:18:05.533231 20990 raft_consensus.cc:1261] T ba83d188a06f40ec9d26d4429031fb63 P 5161fb9c84864bd4bf98f2ce2e724bfa [term 1 FOLLOWER]: Refusing update from remote peer c9ea8357080a4741a2d9e555265ccbe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:18:05.533238 21123 raft_consensus.cc:1261] T ba83d188a06f40ec9d26d4429031fb63 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe [term 1 FOLLOWER]: Refusing update from remote peer c9ea8357080a4741a2d9e555265ccbe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:18:05.534053 21329 consensus_queue.cc:1035] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:18:05.534334 21325 consensus_queue.cc:1035] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:18:05.559020 20846 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:18:05.559279 20846 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:18:05.562783 20846 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a: Sending DeleteTablet for 3 replicas of tablet ba83d188a06f40ec9d26d4429031fb63
I20260926 15:18:05.563606 21236 tablet_service.cc:1514] Processing DeleteTablet for tablet ba83d188a06f40ec9d26d4429031fb63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:18:05 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:35936
I20260926 15:18:05.563715 20970 tablet_service.cc:1514] Processing DeleteTablet for tablet ba83d188a06f40ec9d26d4429031fb63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:18:05 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:45744
I20260926 15:18:05.564009 21103 tablet_service.cc:1514] Processing DeleteTablet for tablet ba83d188a06f40ec9d26d4429031fb63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:18:05 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:39576
I20260926 15:18:05.564888 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 20907
I20260926 15:18:05.574529 21343 tablet_replica.cc:330] T ba83d188a06f40ec9d26d4429031fb63 P 5161fb9c84864bd4bf98f2ce2e724bfa: stopping tablet replica
I20260926 15:18:05.575048 21343 raft_consensus.cc:2229] T ba83d188a06f40ec9d26d4429031fb63 P 5161fb9c84864bd4bf98f2ce2e724bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:18:05.576068 21343 raft_consensus.cc:2258] T ba83d188a06f40ec9d26d4429031fb63 P 5161fb9c84864bd4bf98f2ce2e724bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:18:05.576694 21342 tablet_replica.cc:330] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9: stopping tablet replica
I20260926 15:18:05.576823 21344 tablet_replica.cc:330] T ba83d188a06f40ec9d26d4429031fb63 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe: stopping tablet replica
I20260926 15:18:05.577165 21342 raft_consensus.cc:2229] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:18:05.577212 21344 raft_consensus.cc:2229] T ba83d188a06f40ec9d26d4429031fb63 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:18:05.577847 21342 raft_consensus.cc:2258] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:18:05.578063 21344 raft_consensus.cc:2258] T ba83d188a06f40ec9d26d4429031fb63 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:18:05.578997 21343 ts_tablet_manager.cc:1905] T ba83d188a06f40ec9d26d4429031fb63 P 5161fb9c84864bd4bf98f2ce2e724bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:18:05.579526 21342 ts_tablet_manager.cc:1905] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:18:05.579979 21344 ts_tablet_manager.cc:1905] T ba83d188a06f40ec9d26d4429031fb63 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:18:05.586169 21342 ts_tablet_manager.cc:1918] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:18:05.586369 21342 log.cc:1196] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/ba83d188a06f40ec9d26d4429031fb63
I20260926 15:18:05.586697 21344 ts_tablet_manager.cc:1918] T ba83d188a06f40ec9d26d4429031fb63 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:18:05.586869 21344 log.cc:1196] T ba83d188a06f40ec9d26d4429031fb63 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/ba83d188a06f40ec9d26d4429031fb63
I20260926 15:18:05.586969 21342 ts_tablet_manager.cc:1939] T ba83d188a06f40ec9d26d4429031fb63 P c9ea8357080a4741a2d9e555265ccbe9: Deleting consensus metadata
I20260926 15:18:05.587443 21344 ts_tablet_manager.cc:1939] T ba83d188a06f40ec9d26d4429031fb63 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe: Deleting consensus metadata
I20260926 15:18:05.588676 20831 catalog_manager.cc:4841] TS c9ea8357080a4741a2d9e555265ccbe9 (127.6.209.195:36831): tablet ba83d188a06f40ec9d26d4429031fb63 (table test-table [id=751f90f4c13842ae96a417216fd480e4]) successfully deleted
I20260926 15:18:05.589169 20832 catalog_manager.cc:4841] TS 6c590b9a8f8c4be7a0dee9d3f30eb8fe (127.6.209.194:34839): tablet ba83d188a06f40ec9d26d4429031fb63 (table test-table [id=751f90f4c13842ae96a417216fd480e4]) successfully deleted
Sep 26 15:18:05 dist-test-slave-j6q7 krb5kdc[20801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.193@KRBTEST.COM
I20260926 15:18:05.592494 21343 ts_tablet_manager.cc:1918] T ba83d188a06f40ec9d26d4429031fb63 P 5161fb9c84864bd4bf98f2ce2e724bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:18:05.592674 21343 log.cc:1196] T ba83d188a06f40ec9d26d4429031fb63 P 5161fb9c84864bd4bf98f2ce2e724bfa: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/wal/wals/ba83d188a06f40ec9d26d4429031fb63
I20260926 15:18:05.593209 21343 ts_tablet_manager.cc:1939] T ba83d188a06f40ec9d26d4429031fb63 P 5161fb9c84864bd4bf98f2ce2e724bfa: Deleting consensus metadata
I20260926 15:18:05.594465 20831 catalog_manager.cc:4841] TS 5161fb9c84864bd4bf98f2ce2e724bfa (127.6.209.193:43579): tablet ba83d188a06f40ec9d26d4429031fb63 (table test-table [id=751f90f4c13842ae96a417216fd480e4]) successfully deleted
I20260926 15:18:05.598071 21317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:05.565960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45754 (local address 127.6.209.193:43579)
0926 15:18:05.566154 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:05.566162 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:05.581563 (+ 15401us) server_negotiation.cc:407] Connection header received
0926 15:18:05.581735 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:05.581740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:05.581815 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:05.581920 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:05.583552 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.584592 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.586599 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.587230 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.587338 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:05.591968 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:05.591981 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:05.591995 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:05.592033 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:05.595634 (+ 3601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:05.596189 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:05.596196 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:05.596199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:05.596334 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:05.596666 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:05.596672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:05.596676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:05.597093 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:05.597370 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:05.597550 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 15:18:05.597753 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 15:18:05.598671 21030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:18:05.696728 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 20907
I20260926 15:18:05.713694 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 21040
Sep 26 15:18:05 dist-test-slave-j6q7 krb5kdc[20801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.194@KRBTEST.COM
I20260926 15:18:05.730502 21315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:05.714521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39592 (local address 127.6.209.194:34839)
0926 15:18:05.714739 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:05.714745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:05.714785 (+ 40us) server_negotiation.cc:407] Connection header received
0926 15:18:05.714938 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:05.714943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:05.715048 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:05.715158 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:05.716361 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.717524 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.718736 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.719388 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.719514 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:05.723972 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:05.723991 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:05.724011 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:05.724056 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:05.727768 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:05.728388 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:05.728398 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:05.728401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:05.728562 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:05.728997 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:05.729002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:05.729006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:05.729449 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:05.729747 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:05.729907 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 15:18:05.730130 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 15:18:05.731241 21163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:18:05.825150 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 21040
I20260926 15:18:05.846155 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 21173
Sep 26 15:18:05 dist-test-slave-j6q7 krb5kdc[20801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.195@KRBTEST.COM
I20260926 15:18:05.863171 21314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:05.846992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35958 (local address 127.6.209.195:36831)
0926 15:18:05.847192 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:05.847198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:05.847226 (+ 28us) server_negotiation.cc:407] Connection header received
0926 15:18:05.847363 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:05.847368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:05.847455 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:05.847588 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:05.848899 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.850121 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.851321 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:05.852033 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:05.852171 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:05.856402 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:05.856415 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:05.856436 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:05.856481 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:05.860183 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:05.860959 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:05.860969 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:05.860973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:05.861156 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:05.861511 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:05.861516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:05.861520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:05.862006 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:05.862338 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:05.862496 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 15:18:05.862755 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 15:18:05.863965 21296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:18:05.979436 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 21173
I20260926 15:18:05.997763 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 20814
I20260926 15:18:05.998816 20876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:18:06.094765 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 20814
I20260926 15:18:06.114697 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:40089
--webserver_interface=127.6.209.254
--webserver_port=34359
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:40089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:18:06.400481 21350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:18:06.400861 21350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:18:06.401026 21350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:18:06.411936 21350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:18:06.412072 21350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:18:06.412153 21350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:18:06.412235 21350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:18:06.412305 21350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:18:06.426265 21350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:40089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:40089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=34359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:18:06.427760 21350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:18:06.429319 21350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:18:06.454502 21357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:06.454506 21359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:06.454542 21356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:18:06.455276 21350 server_base.cc:1034] running on GCE node
I20260926 15:18:06.456027 21350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:18:06.457405 21350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:18:06.458621 21350 hybrid_clock.cc:648] HybridClock initialized: now 1790435886458585 us; error 42 us; skew 500 ppm
I20260926 15:18:06.458957 21350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:18:06 dist-test-slave-j6q7 krb5kdc[20801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435886, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:18:06.462237 21350 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:18:06.463696 21350 webserver.cc:460] Webserver started at http://127.6.209.254:34359/ using document root <none> and password file <none>
I20260926 15:18:06.464397 21350 fs_manager.cc:359] Metadata directory not provided
I20260926 15:18:06.464553 21350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:18:06.469852 21350 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:18:06.472432 21366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:06.473716 21350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:18:06.473877 21350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "884244a56c274be9815711313673214a"
format_stamp: "Formatted at 2026-09-26 15:18:03 on dist-test-slave-j6q7"
server_key: "2d6a3b7f7f83efcdb789aed74a14d82c"
server_key_iv: "b211b9142906a90e5312f60a226e1347"
server_key_version: "encryptionkey@0"
I20260926 15:18:06.474848 21350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:18:06.495191 21350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:18:06.499277 21350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:18:06.499550 21350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:18:06.516651 21350 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:40089
I20260926 15:18:06.516669 21417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:40089 every 8 connection(s)
I20260926 15:18:06.518064 21350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:18:06.524380 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 21350
I20260926 15:18:06.524716 21418 sys_catalog.cc:263] Verifying existing consensus state
I20260926 15:18:06.525291 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:43579
--local_ip_for_outbound_sockets=127.6.209.193
--tserver_master_addrs=127.6.209.254:40089
--webserver_port=38465
--webserver_interface=127.6.209.193
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:18:06.529387 21418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a: Bootstrap starting.
I20260926 15:18:06.548444 21418 log.cc:824] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a: Log is configured to *not* fsync() on all Append() calls
I20260926 15:18:06.559734 21418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:18:06.560266 21418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a: Bootstrap complete.
I20260926 15:18:06.568624 21418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884244a56c274be9815711313673214a" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40089 } }
I20260926 15:18:06.569403 21418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 884244a56c274be9815711313673214a, State: Initialized, Role: FOLLOWER
I20260926 15:18:06.569973 21418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884244a56c274be9815711313673214a" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40089 } }
I20260926 15:18:06.570209 21418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:18:06.570351 21418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:18:06.570537 21418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 1 FOLLOWER]: Advancing to term 2
I20260926 15:18:06.574218 21418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884244a56c274be9815711313673214a" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40089 } }
I20260926 15:18:06.574710 21418 leader_election.cc:302] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 884244a56c274be9815711313673214a; no voters:
I20260926 15:18:06.575335 21418 leader_election.cc:288] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 15:18:06.575456 21422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 2 FOLLOWER]: Leader election won for term 2
I20260926 15:18:06.576560 21422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 2 LEADER]: Becoming Leader. State: Replica: 884244a56c274be9815711313673214a, State: Running, Role: LEADER
I20260926 15:18:06.577134 21422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884244a56c274be9815711313673214a" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40089 } }
I20260926 15:18:06.577427 21418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:18:06.579661 21424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 884244a56c274be9815711313673214a. Latest consensus state: current_term: 2 leader_uuid: "884244a56c274be9815711313673214a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884244a56c274be9815711313673214a" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40089 } } }
I20260926 15:18:06.580133 21424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [sys.catalog]: This master's current role is: LEADER
I20260926 15:18:06.580631 21423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "884244a56c274be9815711313673214a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884244a56c274be9815711313673214a" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40089 } } }
I20260926 15:18:06.580973 21429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:18:06.581019 21423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [sys.catalog]: This master's current role is: LEADER
I20260926 15:18:06.594372 21429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:18:06.595598 21429 catalog_manager.cc:1259] Loaded cluster ID: aeea6295143a4ae388e994bdb41048d1
I20260926 15:18:06.595727 21429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:18:06.599418 21429 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 15:18:06.623773 21429 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 15:18:06.636636 21429 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:18:06.637902 21429 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a: Loaded TSK: 0
I20260926 15:18:06.638937 21429 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 15:18:06.853516 21420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:18:06.853745 21420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:18:06.853878 21420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:18:06.859898 21420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:18:06.859975 21420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:18:06.860149 21420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:18:06.868425 21420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:43579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=38465
--tserver_master_addrs=127.6.209.254:40089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:18:06.869522 21420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:18:06.870608 21420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:18:06.882102 21445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:06.882908 21446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:06.883124 21448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:18:06.883451 21420 server_base.cc:1034] running on GCE node
I20260926 15:18:06.883988 21420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:18:06.884784 21420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:18:06.886008 21420 hybrid_clock.cc:648] HybridClock initialized: now 1790435886885959 us; error 48 us; skew 500 ppm
I20260926 15:18:06.886263 21420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:18:06 dist-test-slave-j6q7 krb5kdc[20801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435886, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:18:06.896124 21420 init.cc:377] Logged in from keytab as kudu/127.6.209.193@KRBTEST.COM (short username kudu)
I20260926 15:18:06.897588 21420 webserver.cc:460] Webserver started at http://127.6.209.193:38465/ using document root <none> and password file <none>
I20260926 15:18:06.898185 21420 fs_manager.cc:359] Metadata directory not provided
I20260926 15:18:06.898322 21420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:18:06.903640 21420 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:18:06.905930 21455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:06.907068 21420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:18:06.907204 21420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "5161fb9c84864bd4bf98f2ce2e724bfa"
format_stamp: "Formatted at 2026-09-26 15:18:04 on dist-test-slave-j6q7"
server_key: "79d81f1983a94a17bc9f9518b67a8f55"
server_key_iv: "66bef5cd196840de3154e2e823da9791"
server_key_version: "encryptionkey@0"
I20260926 15:18:06.907990 21420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:18:06.928898 21420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:18:06.932595 21420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:18:06.932943 21420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:18:06.933844 21420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:18:06.935075 21420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:18:06.935158 21420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:06.935280 21420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:18:06.935365 21420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:06.966553 21420 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:43579
I20260926 15:18:06.966651 21567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:43579 every 8 connection(s)
I20260926 15:18:06.967887 21420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:18:06.977730 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 21420
I20260926 15:18:06.978299 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:34839
--local_ip_for_outbound_sockets=127.6.209.194
--tserver_master_addrs=127.6.209.254:40089
--webserver_port=33857
--webserver_interface=127.6.209.194
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:18:06 dist-test-slave-j6q7 krb5kdc[20801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435886, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:18:06.988590 21571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:06.970439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.193:36421)
0926 15:18:06.971122 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:06.971161 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:06.972878 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:06.974007 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:06.974014 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:06.974467 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:18:06.974911 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:06.974926 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:06.976248 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:06.976251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:06.977475 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:06.977491 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:06.978515 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:06.978523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:06.978768 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:06.979408 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:18:06.979436 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:18:06.981974 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:18:06.984997 (+ 3023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:18:06.985007 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:18:06.985018 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:18:06.985412 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:18:06.985796 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:18:06.985804 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:18:06.985807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:18:06.985977 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:18:06.986882 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:18:06.986889 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:18:06.987255 (+ 366us) client_negotiation.cc:769] Sending connection context
0926 15:18:06.987448 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 15:18:06.987756 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":286,"threads_started":1}
I20260926 15:18:06.989156 21570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:06.970285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:36421 (local address 127.6.209.254:40089)
0926 15:18:06.970921 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:06.970931 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:06.971268 (+ 337us) server_negotiation.cc:407] Connection header received
0926 15:18:06.973376 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:06.973387 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:06.973653 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:06.973895 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:06.975082 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:06.976130 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:06.977593 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:06.978293 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:06.978517 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:06.982217 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:06.982224 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:06.982236 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:06.982273 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:06.984796 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:06.985543 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:06.985549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:06.985554 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:06.985693 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:06.986125 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:06.986131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:06.986135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:06.986711 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:06.986868 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:06.987674 (+ 806us) server_negotiation.cc:299] Negotiation successful
0926 15:18:06.988052 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":299,"threads_started":1}
I20260926 15:18:06.991533 21568 heartbeater.cc:347] Connected to a master server at 127.6.209.254:40089
I20260926 15:18:06.991827 21568 heartbeater.cc:464] Registering TS with master...
I20260926 15:18:06.992619 21568 heartbeater.cc:511] Master 127.6.209.254:40089 requested a full tablet report, sending...
I20260926 15:18:06.994674 21383 ts_manager.cc:194] Registered new tserver with Master: 5161fb9c84864bd4bf98f2ce2e724bfa (127.6.209.193:43579)
I20260926 15:18:06.996106 21383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.193@KRBTEST.COM'} at 127.6.209.193:36421
W20260926 15:18:07.267969 21572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:18:07.268177 21572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:18:07.268313 21572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:18:07.274477 21572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:18:07.274585 21572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:18:07.274775 21572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:18:07.283238 21572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:34839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=33857
--tserver_master_addrs=127.6.209.254:40089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:18:07.284225 21572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:18:07.285336 21572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:18:07.296723 21578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:07.296702 21580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:07.296849 21577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:18:07.297367 21572 server_base.cc:1034] running on GCE node
I20260926 15:18:07.297880 21572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:18:07.298969 21572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:18:07.300227 21572 hybrid_clock.cc:648] HybridClock initialized: now 1790435887300194 us; error 31 us; skew 500 ppm
I20260926 15:18:07.300535 21572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:18:07 dist-test-slave-j6q7 krb5kdc[20801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435887, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:18:07.303400 21572 init.cc:377] Logged in from keytab as kudu/127.6.209.194@KRBTEST.COM (short username kudu)
I20260926 15:18:07.304651 21572 webserver.cc:460] Webserver started at http://127.6.209.194:33857/ using document root <none> and password file <none>
I20260926 15:18:07.305296 21572 fs_manager.cc:359] Metadata directory not provided
I20260926 15:18:07.305403 21572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:18:07.309722 21572 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 15:18:07.312163 21587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:07.313330 21572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:18:07.313472 21572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe"
format_stamp: "Formatted at 2026-09-26 15:18:04 on dist-test-slave-j6q7"
server_key: "7a8dc3e8c6ae79ba4032bc55343ce88b"
server_key_iv: "b5fc3a54d08c31557fa0a1bda370fd88"
server_key_version: "encryptionkey@0"
I20260926 15:18:07.314239 21572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:18:07.337002 21572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:18:07.339957 21572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:18:07.340229 21572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:18:07.341097 21572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:18:07.342257 21572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:18:07.342365 21572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:07.342482 21572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:18:07.342571 21572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:07.372344 21572 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:34839
I20260926 15:18:07.372427 21699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:34839 every 8 connection(s)
I20260926 15:18:07.373561 21572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:18:07.381927 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 21572
I20260926 15:18:07.382563 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:36831
--local_ip_for_outbound_sockets=127.6.209.195
--tserver_master_addrs=127.6.209.254:40089
--webserver_port=36133
--webserver_interface=127.6.209.195
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:18:07 dist-test-slave-j6q7 krb5kdc[20801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435887, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:18:07.392396 21570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:07.375635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:55869 (local address 127.6.209.254:40089)
0926 15:18:07.375868 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:07.375873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:07.376760 (+ 887us) server_negotiation.cc:407] Connection header received
0926 15:18:07.378432 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:07.378436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:07.378495 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:07.378569 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:07.379763 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.380391 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.382159 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.382577 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.382715 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:07.386494 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:07.386498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:07.386501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:07.386537 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:07.388675 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:07.389793 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:07.389797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:07.389799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:07.389957 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:07.390442 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:07.390445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:07.390447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:07.390837 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:07.390991 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:07.391707 (+ 716us) server_negotiation.cc:299] Negotiation successful
0926 15:18:07.391966 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 15:18:07.392802 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:07.376048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.194:55869)
0926 15:18:07.376616 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:07.376651 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:07.378201 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:07.378735 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:07.378742 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:07.379190 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:18:07.379633 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:07.379648 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.380512 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.380515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:07.382015 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:07.382031 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.382816 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.382825 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:07.383075 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:07.383724 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:18:07.383752 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:18:07.386327 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:18:07.388804 (+ 2477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:18:07.388994 (+ 190us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:18:07.389105 (+ 111us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:18:07.389666 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:18:07.390069 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:18:07.390076 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:18:07.390082 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:18:07.390321 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:18:07.390997 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:18:07.391004 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:18:07.391332 (+ 328us) client_negotiation.cc:769] Sending connection context
0926 15:18:07.391535 (+ 203us) client_negotiation.cc:240] Negotiation successful
0926 15:18:07.391854 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":225,"threads_started":1}
I20260926 15:18:07.394025 21700 heartbeater.cc:347] Connected to a master server at 127.6.209.254:40089
I20260926 15:18:07.394254 21700 heartbeater.cc:464] Registering TS with master...
I20260926 15:18:07.394826 21700 heartbeater.cc:511] Master 127.6.209.254:40089 requested a full tablet report, sending...
I20260926 15:18:07.396095 21383 ts_manager.cc:194] Registered new tserver with Master: 6c590b9a8f8c4be7a0dee9d3f30eb8fe (127.6.209.194:34839)
I20260926 15:18:07.396870 21383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.194@KRBTEST.COM'} at 127.6.209.194:55869
W20260926 15:18:07.673806 21703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:18:07.674009 21703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:18:07.674149 21703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:18:07.680558 21703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:18:07.680668 21703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:18:07.680893 21703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:18:07.689739 21703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:36831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=36133
--tserver_master_addrs=127.6.209.254:40089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:18:07.690747 21703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:18:07.692044 21703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:18:07.704612 21709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:07.704694 21711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:18:07.705292 21703 server_base.cc:1034] running on GCE node
W20260926 15:18:07.705564 21708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:18:07.706116 21703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:18:07.706897 21703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:18:07.708128 21703 hybrid_clock.cc:648] HybridClock initialized: now 1790435887708103 us; error 38 us; skew 500 ppm
I20260926 15:18:07.708423 21703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:18:07 dist-test-slave-j6q7 krb5kdc[20801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435887, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:18:07.711181 21703 init.cc:377] Logged in from keytab as kudu/127.6.209.195@KRBTEST.COM (short username kudu)
I20260926 15:18:07.712437 21703 webserver.cc:460] Webserver started at http://127.6.209.195:36133/ using document root <none> and password file <none>
I20260926 15:18:07.713127 21703 fs_manager.cc:359] Metadata directory not provided
I20260926 15:18:07.713224 21703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:18:07.717548 21703 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:18:07.719821 21718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:07.721028 21703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:18:07.721160 21703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "c9ea8357080a4741a2d9e555265ccbe9"
format_stamp: "Formatted at 2026-09-26 15:18:05 on dist-test-slave-j6q7"
server_key: "05960ef35ec7dfe6e78a3e2bf9a5c885"
server_key_iv: "d41a0ce4f35f8a07ee2a75186c4739b2"
server_key_version: "encryptionkey@0"
I20260926 15:18:07.721992 21703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:18:07.742534 21703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:18:07.745811 21703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:18:07.745990 21703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:18:07.746737 21703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:18:07.747792 21703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:18:07.747887 21703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:07.747968 21703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:18:07.748020 21703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:07.777463 21703 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:36831
I20260926 15:18:07.777529 21830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:36831 every 8 connection(s)
I20260926 15:18:07.778834 21703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:18:07.783972 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 21703
Sep 26 15:18:07 dist-test-slave-j6q7 krb5kdc[20801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435887, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:18:07.797363 21570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:07.781077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:48911 (local address 127.6.209.254:40089)
0926 15:18:07.781249 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:07.781254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:07.782389 (+ 1135us) server_negotiation.cc:407] Connection header received
0926 15:18:07.783965 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:07.783970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:07.784046 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:07.784155 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:07.785793 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.786561 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.788154 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.788530 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.788604 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:07.791874 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:07.791880 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:07.791884 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:07.791923 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:07.794677 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:07.795422 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:07.795425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:07.795427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:07.795565 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:07.795932 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:07.795937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:07.795939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:07.796161 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:07.796291 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:07.796896 (+ 605us) server_negotiation.cc:299] Negotiation successful
0926 15:18:07.797072 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 15:18:07.798074 21833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:07.781579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.195:48911)
0926 15:18:07.782238 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:07.782277 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:07.783723 (+ 1446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:07.784356 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:07.784363 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:07.784931 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:18:07.785651 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:07.785673 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.786702 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.786708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:07.788034 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:07.788044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.788720 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.788727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:07.788978 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:07.789532 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:18:07.789568 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:18:07.791686 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:18:07.794808 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:18:07.794817 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:18:07.794830 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:18:07.795320 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:18:07.795674 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:18:07.795679 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:18:07.795682 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:18:07.795828 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:18:07.796272 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:18:07.796278 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:18:07.796521 (+ 243us) client_negotiation.cc:769] Sending connection context
0926 15:18:07.796790 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 15:18:07.797199 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":244,"threads_started":1}
I20260926 15:18:07.798548 21835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:07.784731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37280 (local address 127.6.209.254:40089)
0926 15:18:07.785149 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:07.785154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:07.785639 (+ 485us) server_negotiation.cc:407] Connection header received
0926 15:18:07.785793 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:07.785797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:07.785860 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:07.785982 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:07.787475 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.788107 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.789595 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.790078 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.790219 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:07.792644 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:07.792648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:07.792651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:07.792687 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:07.795544 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:07.796331 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:07.796339 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:07.796341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:07.796506 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:07.796903 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:07.796908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:07.796910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:07.797144 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:07.797374 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:07.798100 (+ 726us) server_negotiation.cc:299] Negotiation successful
0926 15:18:07.798310 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":229,"threads_started":1}
I20260926 15:18:07.799304 21831 heartbeater.cc:347] Connected to a master server at 127.6.209.254:40089
I20260926 15:18:07.799576 21831 heartbeater.cc:464] Registering TS with master...
I20260926 15:18:07.800252 21831 heartbeater.cc:511] Master 127.6.209.254:40089 requested a full tablet report, sending...
I20260926 15:18:07.801687 21382 ts_manager.cc:194] Registered new tserver with Master: c9ea8357080a4741a2d9e555265ccbe9 (127.6.209.195:36831)
I20260926 15:18:07.802526 21382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.195@KRBTEST.COM'} at 127.6.209.195:48911
I20260926 15:18:07.812631 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:18:07.817970 21835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:07.813419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37286 (local address 127.6.209.254:40089)
0926 15:18:07.813539 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:07.813542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:07.813985 (+ 443us) server_negotiation.cc:407] Connection header received
0926 15:18:07.814109 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:07.814112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:07.814204 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:07.814298 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:18:07.814609 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.815144 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.816359 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.816745 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.816868 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:07.817231 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:18:07.817594 (+ 363us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:18:07.817710 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:07.817744 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 15:18:07.817788 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 15:18:07.818786 21382 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:37286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:18:07.825619 21382 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:18:07.846733 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:07.837247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39598 (local address 127.6.209.194:34839)
0926 15:18:07.839426 (+ 2179us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:07.839433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:07.839462 (+ 29us) server_negotiation.cc:407] Connection header received
0926 15:18:07.839543 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:07.839550 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:07.839809 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:07.839988 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:18:07.841084 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.842562 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.844857 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.845710 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.845844 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:07.846068 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:07.846295 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 15:18:07.846434 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1998,"thread_start_us":1852,"threads_started":1}
I20260926 15:18:07.847188 21841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:07.835693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:43579 (local address 127.0.0.1:45760)
0926 15:18:07.836488 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:07.836549 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:07.836719 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:07.840192 (+ 3473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:07.840195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:07.840228 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:18:07.840781 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:07.840792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.842393 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.842400 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:07.844693 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:07.844709 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.846423 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.846428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:07.846582 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:07.846622 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 15:18:07.846688 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 15:18:07.846807 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":47,"spinlock_wait_cycles":8832,"thread_start_us":381,"threads_started":1}
I20260926 15:18:07.846825 21843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:07.836147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:34839 (local address 127.0.0.1:39598)
0926 15:18:07.838701 (+ 2554us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:07.838726 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:07.838876 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:07.840281 (+ 1405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:07.840287 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:07.840341 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:18:07.840882 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:07.840898 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.842703 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.842708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:07.844703 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:07.844713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.845886 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.845891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:07.846073 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:07.846130 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 15:18:07.846259 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 15:18:07.846417 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2395,"mutex_wait_us":2,"spinlock_wait_cycles":24960,"thread_start_us":2245,"threads_started":1}
I20260926 15:18:07.847747 21846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:07.837186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35960 (local address 127.6.209.195:36831)
0926 15:18:07.838167 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:07.838177 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:07.839156 (+ 979us) server_negotiation.cc:407] Connection header received
0926 15:18:07.839439 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:07.839448 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:07.839651 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:07.839824 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:18:07.841493 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.842933 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.844959 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.846209 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.846344 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:07.846539 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:07.847271 (+ 732us) server_negotiation.cc:299] Negotiation successful
0926 15:18:07.847427 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":781,"thread_start_us":650,"threads_started":1}
I20260926 15:18:07.847856 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:07.835739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45760 (local address 127.6.209.193:43579)
0926 15:18:07.839627 (+ 3888us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:07.839633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:07.839656 (+ 23us) server_negotiation.cc:407] Connection header received
0926 15:18:07.839763 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:07.839771 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:07.839976 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:07.840289 (+ 313us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:18:07.841144 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.842259 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.844927 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.846209 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.846921 (+ 712us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:07.847104 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:07.847162 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 15:18:07.847548 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3753,"thread_start_us":3644,"threads_started":1}
I20260926 15:18:07.848564 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:07.836853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:36831 (local address 127.0.0.1:35960)
0926 15:18:07.839038 (+ 2185us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:07.839059 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:07.839255 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:07.840176 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:07.840183 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:07.840222 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:18:07.841303 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:07.841314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.843057 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.843062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:07.844797 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:07.844811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.846682 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.846688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:07.846855 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:07.846897 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 15:18:07.846958 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 15:18:07.847055 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2046,"mutex_wait_us":22,"spinlock_wait_cycles":768,"thread_start_us":1901,"threads_started":1}
I20260926 15:18:07.850765 21503 tablet_service.cc:1467] Processing CreateTablet for tablet a133654cf3d74149949f8d8a30ceabe5 (DEFAULT_TABLE table=test-table [id=b14581510955423aa0f92f023e05a6c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:18:07.850869 21635 tablet_service.cc:1467] Processing CreateTablet for tablet a133654cf3d74149949f8d8a30ceabe5 (DEFAULT_TABLE table=test-table [id=b14581510955423aa0f92f023e05a6c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:18:07.851971 21503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a133654cf3d74149949f8d8a30ceabe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:18:07.851611 21766 tablet_service.cc:1467] Processing CreateTablet for tablet a133654cf3d74149949f8d8a30ceabe5 (DEFAULT_TABLE table=test-table [id=b14581510955423aa0f92f023e05a6c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:18:07.852202 21635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a133654cf3d74149949f8d8a30ceabe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:18:07.852813 21766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a133654cf3d74149949f8d8a30ceabe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:18:07.862344 21847 tablet_bootstrap.cc:492] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa: Bootstrap starting.
I20260926 15:18:07.865044 21847 tablet_bootstrap.cc:654] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa: Neither blocks nor log segments found. Creating new log.
I20260926 15:18:07.865764 21849 tablet_bootstrap.cc:492] T a133654cf3d74149949f8d8a30ceabe5 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe: Bootstrap starting.
I20260926 15:18:07.867223 21847 log.cc:824] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa: Log is configured to *not* fsync() on all Append() calls
I20260926 15:18:07.868508 21848 tablet_bootstrap.cc:492] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9: Bootstrap starting.
I20260926 15:18:07.868937 21849 tablet_bootstrap.cc:654] T a133654cf3d74149949f8d8a30ceabe5 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe: Neither blocks nor log segments found. Creating new log.
I20260926 15:18:07.869791 21847 tablet_bootstrap.cc:492] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa: No bootstrap required, opened a new log
I20260926 15:18:07.870034 21847 ts_tablet_manager.cc:1397] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 15:18:07.870038 21849 log.cc:824] T a133654cf3d74149949f8d8a30ceabe5 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe: Log is configured to *not* fsync() on all Append() calls
I20260926 15:18:07.871624 21848 tablet_bootstrap.cc:654] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9: Neither blocks nor log segments found. Creating new log.
I20260926 15:18:07.872819 21849 tablet_bootstrap.cc:492] T a133654cf3d74149949f8d8a30ceabe5 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe: No bootstrap required, opened a new log
I20260926 15:18:07.872763 21848 log.cc:824] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9: Log is configured to *not* fsync() on all Append() calls
I20260926 15:18:07.873060 21849 ts_tablet_manager.cc:1397] T a133654cf3d74149949f8d8a30ceabe5 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260926 15:18:07.876670 21847 raft_consensus.cc:348] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43579 } } peers { permanent_uuid: "c9ea8357080a4741a2d9e555265ccbe9" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36831 } } peers { permanent_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34839 } }
I20260926 15:18:07.877118 21847 raft_consensus.cc:374] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:18:07.877213 21847 raft_consensus.cc:729] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5161fb9c84864bd4bf98f2ce2e724bfa, State: Initialized, Role: FOLLOWER
I20260926 15:18:07.877835 21847 consensus_queue.cc:260] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43579 } } peers { permanent_uuid: "c9ea8357080a4741a2d9e555265ccbe9" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36831 } } peers { permanent_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34839 } }
I20260926 15:18:07.878829 21568 heartbeater.cc:503] Master 127.6.209.254:40089 was elected leader, sending a full tablet report...
I20260926 15:18:07.879456 21847 ts_tablet_manager.cc:1428] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa: Time spent starting tablet: real 0.009s user 0.006s sys 0.004s
I20260926 15:18:07.879493 21848 tablet_bootstrap.cc:492] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9: No bootstrap required, opened a new log
I20260926 15:18:07.879751 21848 ts_tablet_manager.cc:1397] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260926 15:18:07.883134 21849 raft_consensus.cc:348] T a133654cf3d74149949f8d8a30ceabe5 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43579 } } peers { permanent_uuid: "c9ea8357080a4741a2d9e555265ccbe9" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36831 } } peers { permanent_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34839 } }
I20260926 15:18:07.883550 21849 raft_consensus.cc:374] T a133654cf3d74149949f8d8a30ceabe5 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:18:07.883636 21849 raft_consensus.cc:729] T a133654cf3d74149949f8d8a30ceabe5 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c590b9a8f8c4be7a0dee9d3f30eb8fe, State: Initialized, Role: FOLLOWER
I20260926 15:18:07.884114 21849 consensus_queue.cc:260] T a133654cf3d74149949f8d8a30ceabe5 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43579 } } peers { permanent_uuid: "c9ea8357080a4741a2d9e555265ccbe9" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36831 } } peers { permanent_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34839 } }
I20260926 15:18:07.884980 21700 heartbeater.cc:503] Master 127.6.209.254:40089 was elected leader, sending a full tablet report...
I20260926 15:18:07.885488 21849 ts_tablet_manager.cc:1428] T a133654cf3d74149949f8d8a30ceabe5 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
I20260926 15:18:07.888285 21848 raft_consensus.cc:348] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43579 } } peers { permanent_uuid: "c9ea8357080a4741a2d9e555265ccbe9" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36831 } } peers { permanent_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34839 } }
I20260926 15:18:07.888644 21848 raft_consensus.cc:374] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:18:07.888748 21848 raft_consensus.cc:729] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ea8357080a4741a2d9e555265ccbe9, State: Initialized, Role: FOLLOWER
I20260926 15:18:07.889183 21848 consensus_queue.cc:260] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43579 } } peers { permanent_uuid: "c9ea8357080a4741a2d9e555265ccbe9" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36831 } } peers { permanent_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34839 } }
I20260926 15:18:07.889936 21831 heartbeater.cc:503] Master 127.6.209.254:40089 was elected leader, sending a full tablet report...
I20260926 15:18:07.890385 21848 ts_tablet_manager.cc:1428] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
W20260926 15:18:07.970000 21569 tablet.cc:2369] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:18:07.987195 21853 raft_consensus.cc:482] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:18:07.987424 21853 raft_consensus.cc:504] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43579 } } peers { permanent_uuid: "c9ea8357080a4741a2d9e555265ccbe9" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36831 } } peers { permanent_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34839 } }
I20260926 15:18:07.988772 21853 leader_election.cc:288] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9ea8357080a4741a2d9e555265ccbe9 (127.6.209.195:36831), 6c590b9a8f8c4be7a0dee9d3f30eb8fe (127.6.209.194:34839)
I20260926 15:18:07.993914 21855 raft_consensus.cc:482] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:18:07.994207 21855 raft_consensus.cc:504] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43579 } } peers { permanent_uuid: "c9ea8357080a4741a2d9e555265ccbe9" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36831 } } peers { permanent_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34839 } }
I20260926 15:18:07.995626 21846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:07.989255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:41887 (local address 127.6.209.195:36831)
0926 15:18:07.989428 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:07.989435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:07.990126 (+ 691us) server_negotiation.cc:407] Connection header received
0926 15:18:07.990336 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:07.990341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:07.990405 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:07.990494 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:18:07.991022 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.991727 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.993636 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.994690 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.994879 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:07.994956 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:07.995271 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 15:18:07.995381 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 15:18:07.995724 21856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:07.989006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:36831 (local address 127.6.209.193:41887)
0926 15:18:07.990029 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:07.990041 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:07.990215 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:07.990508 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:07.990511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:07.990529 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:18:07.990869 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:07.990878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.991858 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.991861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:07.993378 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:07.993407 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.994906 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.994911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:07.995102 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:07.995150 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 15:18:07.995249 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 15:18:07.995368 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"mutex_wait_us":65,"spinlock_wait_cycles":768,"thread_start_us":409,"threads_started":1}
I20260926 15:18:07.995831 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:07.989110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:51097 (local address 127.6.209.194:34839)
0926 15:18:07.989292 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:07.989298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:07.989771 (+ 473us) server_negotiation.cc:407] Connection header received
0926 15:18:07.989967 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:07.989972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:07.990029 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:07.990109 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:18:07.990722 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.991542 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.993457 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.994715 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.994875 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:07.994962 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:07.995499 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 15:18:07.995614 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 15:18:07.996518 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:07.989006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:34839 (local address 127.6.209.193:51097)
0926 15:18:07.989671 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:07.989684 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:07.989840 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:07.990125 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:07.990128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:07.990211 (+ 83us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:18:07.990560 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:07.990576 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.991703 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.991708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:07.993327 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:07.993340 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.995106 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.995112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:07.995324 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:07.995354 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 15:18:07.995988 (+ 634us) client_negotiation.cc:240] Negotiation successful
0926 15:18:07.996122 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":49,"spinlock_wait_cycles":100608,"thread_start_us":786,"threads_started":1}
I20260926 15:18:07.997087 21786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a133654cf3d74149949f8d8a30ceabe5" candidate_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ea8357080a4741a2d9e555265ccbe9" is_pre_election: true
I20260926 15:18:07.997538 21786 raft_consensus.cc:2454] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5161fb9c84864bd4bf98f2ce2e724bfa in term 0.
I20260926 15:18:07.997622 21855 leader_election.cc:288] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5161fb9c84864bd4bf98f2ce2e724bfa (127.6.209.193:43579), 6c590b9a8f8c4be7a0dee9d3f30eb8fe (127.6.209.194:34839)
I20260926 15:18:07.998276 21655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a133654cf3d74149949f8d8a30ceabe5" candidate_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" is_pre_election: true
I20260926 15:18:07.998646 21655 raft_consensus.cc:2454] T a133654cf3d74149949f8d8a30ceabe5 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5161fb9c84864bd4bf98f2ce2e724bfa in term 0.
I20260926 15:18:07.998656 21457 leader_election.cc:302] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5161fb9c84864bd4bf98f2ce2e724bfa, c9ea8357080a4741a2d9e555265ccbe9; no voters:
I20260926 15:18:07.999171 21853 raft_consensus.cc:2781] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:18:07.999311 21853 raft_consensus.cc:482] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:18:07.999404 21853 raft_consensus.cc:3037] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:18:08.002218 21853 raft_consensus.cc:504] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43579 } } peers { permanent_uuid: "c9ea8357080a4741a2d9e555265ccbe9" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36831 } } peers { permanent_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34839 } }
I20260926 15:18:08.002593 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:07.996789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:41681 (local address 127.6.209.193:43579)
0926 15:18:07.996981 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:07.996989 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:07.997010 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:18:07.997071 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:07.997077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:07.997155 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:07.997250 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:18:07.997733 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.998660 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:08.000948 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:08.001741 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:08.001892 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:08.001953 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:08.002235 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 15:18:08.002348 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 15:18:08.002611 21833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:07.996175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:43579 (local address 127.6.209.195:41681)
0926 15:18:07.996352 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:07.996374 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:07.996471 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:07.997363 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:07.997367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:07.997394 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:18:07.997566 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:07.997578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:07.998831 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:07.998836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:08.000781 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:08.000792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:08.001885 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:08.001890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:08.002081 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:08.002106 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 15:18:08.002209 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 15:18:08.002334 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":4}
I20260926 15:18:08.003083 21853 leader_election.cc:288] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa [CANDIDATE]: Term 1 election: Requested vote from peers c9ea8357080a4741a2d9e555265ccbe9 (127.6.209.195:36831), 6c590b9a8f8c4be7a0dee9d3f30eb8fe (127.6.209.194:34839)
I20260926 15:18:08.003445 21523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a133654cf3d74149949f8d8a30ceabe5" candidate_uuid: "c9ea8357080a4741a2d9e555265ccbe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" is_pre_election: true
I20260926 15:18:08.003530 21655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a133654cf3d74149949f8d8a30ceabe5" candidate_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe"
I20260926 15:18:08.003728 21655 raft_consensus.cc:3037] T a133654cf3d74149949f8d8a30ceabe5 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:18:08.003845 21523 raft_consensus.cc:2379] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9ea8357080a4741a2d9e555265ccbe9 in current term 1: Already voted for candidate 5161fb9c84864bd4bf98f2ce2e724bfa in this term.
I20260926 15:18:08.005174 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:07.996538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:54631 (local address 127.6.209.194:34839)
0926 15:18:07.997511 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:07.997519 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:07.998251 (+ 732us) server_negotiation.cc:407] Connection header received
0926 15:18:07.998480 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:07.998486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:07.998568 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:07.998636 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:18:07.999466 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:08.000489 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:08.002904 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:08.004106 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:08.004228 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:08.004292 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:08.004799 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 15:18:08.004941 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":830}
I20260926 15:18:08.005242 21858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:07.996223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:34839 (local address 127.6.209.195:54631)
0926 15:18:07.998177 (+ 1954us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:07.998191 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:07.998339 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:07.998806 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:07.998810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:07.998831 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:18:07.999298 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:07.999312 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:08.000635 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:08.000642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:08.002640 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:08.002652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:08.004479 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:08.004485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:08.004646 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:08.004671 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 15:18:08.004756 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 15:18:08.004897 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1851,"mutex_wait_us":562,"thread_start_us":1180,"threads_started":1}
I20260926 15:18:08.005193 21786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a133654cf3d74149949f8d8a30ceabe5" candidate_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ea8357080a4741a2d9e555265ccbe9"
I20260926 15:18:08.005380 21786 raft_consensus.cc:3037] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:18:08.005920 21654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a133654cf3d74149949f8d8a30ceabe5" candidate_uuid: "c9ea8357080a4741a2d9e555265ccbe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" is_pre_election: true
I20260926 15:18:08.006776 21655 raft_consensus.cc:2454] T a133654cf3d74149949f8d8a30ceabe5 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5161fb9c84864bd4bf98f2ce2e724bfa in term 1.
I20260926 15:18:08.006713 21721 leader_election.cc:302] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9ea8357080a4741a2d9e555265ccbe9; no voters: 5161fb9c84864bd4bf98f2ce2e724bfa, 6c590b9a8f8c4be7a0dee9d3f30eb8fe
I20260926 15:18:08.007164 21458 leader_election.cc:302] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5161fb9c84864bd4bf98f2ce2e724bfa, 6c590b9a8f8c4be7a0dee9d3f30eb8fe; no voters:
I20260926 15:18:08.007428 21853 raft_consensus.cc:2781] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:18:08.007797 21853 raft_consensus.cc:686] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa [term 1 LEADER]: Becoming Leader. State: Replica: 5161fb9c84864bd4bf98f2ce2e724bfa, State: Running, Role: LEADER
I20260926 15:18:08.007939 21786 raft_consensus.cc:2454] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5161fb9c84864bd4bf98f2ce2e724bfa in term 1.
I20260926 15:18:08.008070 21855 raft_consensus.cc:2726] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 15:18:08.008350 21853 consensus_queue.cc:237] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43579 } } peers { permanent_uuid: "c9ea8357080a4741a2d9e555265ccbe9" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36831 } } peers { permanent_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34839 } }
I20260926 15:18:08.012657 21382 catalog_manager.cc:5495] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa reported cstate change: term changed from 0 to 1, leader changed from <none> to 5161fb9c84864bd4bf98f2ce2e724bfa (127.6.209.193). New cstate: current_term: 1 leader_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5161fb9c84864bd4bf98f2ce2e724bfa" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9ea8357080a4741a2d9e555265ccbe9" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34839 } health_report { overall_health: UNKNOWN } } }
W20260926 15:18:08.030181 21832 tablet.cc:2369] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:18:08.033058 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:08.028398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45776 (local address 127.6.209.193:43579)
0926 15:18:08.028531 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:08.028535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:08.028601 (+ 66us) server_negotiation.cc:407] Connection header received
0926 15:18:08.028741 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:08.028744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:08.028861 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:08.028972 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:18:08.029269 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:08.029920 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:08.031241 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:08.031618 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:08.031710 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:08.032107 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:18:08.032528 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:18:08.032627 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:08.032696 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 15:18:08.032749 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 15:18:08.039310 21786 raft_consensus.cc:1261] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9 [term 1 FOLLOWER]: Refusing update from remote peer 5161fb9c84864bd4bf98f2ce2e724bfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:18:08.039346 21655 raft_consensus.cc:1261] T a133654cf3d74149949f8d8a30ceabe5 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe [term 1 FOLLOWER]: Refusing update from remote peer 5161fb9c84864bd4bf98f2ce2e724bfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:18:08.040078 21859 consensus_queue.cc:1035] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ea8357080a4741a2d9e555265ccbe9" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:18:08.040429 21853 consensus_queue.cc:1035] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:18:08.061189 21382 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:37286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:18:08.061410 21382 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:37286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:18:08.064584 21382 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a: Sending DeleteTablet for 3 replicas of tablet a133654cf3d74149949f8d8a30ceabe5
I20260926 15:18:08.065622 21635 tablet_service.cc:1514] Processing DeleteTablet for tablet a133654cf3d74149949f8d8a30ceabe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:18:08 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:39598
I20260926 15:18:08.065752 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 21420
I20260926 15:18:08.065811 21503 tablet_service.cc:1514] Processing DeleteTablet for tablet a133654cf3d74149949f8d8a30ceabe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:18:08 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:45760
I20260926 15:18:08.066325 21766 tablet_service.cc:1514] Processing DeleteTablet for tablet a133654cf3d74149949f8d8a30ceabe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:18:08 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:35960
I20260926 15:18:08.074966 21873 tablet_replica.cc:330] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9: stopping tablet replica
I20260926 15:18:08.075752 21873 raft_consensus.cc:2229] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:18:08.076573 21873 raft_consensus.cc:2258] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:18:08.079540 21873 ts_tablet_manager.cc:1905] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:18:08.081940 21872 tablet_replica.cc:330] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa: stopping tablet replica
I20260926 15:18:08.082484 21872 raft_consensus.cc:2229] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:18:08.083135 21872 raft_consensus.cc:2258] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:18:08.084700 21872 ts_tablet_manager.cc:1905] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:18:08.086850 21873 ts_tablet_manager.cc:1918] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:18:08.086912 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:08.067750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45788 (local address 127.6.209.193:43579)
0926 15:18:08.067970 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:08.067977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:08.067995 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:18:08.068050 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:08.068057 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:08.068134 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:08.068232 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:08.070312 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:08.071723 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:08.074127 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:08.074888 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:08.074997 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:08.078869 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:08.078885 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:08.078902 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:08.078951 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:08.083173 (+ 4222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:08.083996 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:08.084002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:08.084005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:08.084164 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:08.084605 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:08.084611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:08.084618 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:08.085027 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:08.085341 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:08.086329 (+ 988us) server_negotiation.cc:299] Negotiation successful
0926 15:18:08.086538 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 15:18:08.087249 21873 log.cc:1196] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/a133654cf3d74149949f8d8a30ceabe5
I20260926 15:18:08.087494 21563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:18:08.087795 21871 tablet_replica.cc:330] T a133654cf3d74149949f8d8a30ceabe5 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe: stopping tablet replica
I20260926 15:18:08.087834 21873 ts_tablet_manager.cc:1939] T a133654cf3d74149949f8d8a30ceabe5 P c9ea8357080a4741a2d9e555265ccbe9: Deleting consensus metadata
I20260926 15:18:08.088212 21871 raft_consensus.cc:2229] T a133654cf3d74149949f8d8a30ceabe5 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:18:08.088691 21871 raft_consensus.cc:2258] T a133654cf3d74149949f8d8a30ceabe5 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:18:08.089790 21368 catalog_manager.cc:4841] TS c9ea8357080a4741a2d9e555265ccbe9 (127.6.209.195:36831): tablet a133654cf3d74149949f8d8a30ceabe5 (table test-table [id=b14581510955423aa0f92f023e05a6c0]) successfully deleted
I20260926 15:18:08.090885 21871 ts_tablet_manager.cc:1905] T a133654cf3d74149949f8d8a30ceabe5 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:18:08.097373 21871 ts_tablet_manager.cc:1918] T a133654cf3d74149949f8d8a30ceabe5 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:18:08.097568 21871 log.cc:1196] T a133654cf3d74149949f8d8a30ceabe5 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/a133654cf3d74149949f8d8a30ceabe5
I20260926 15:18:08.098029 21871 ts_tablet_manager.cc:1939] T a133654cf3d74149949f8d8a30ceabe5 P 6c590b9a8f8c4be7a0dee9d3f30eb8fe: Deleting consensus metadata
I20260926 15:18:08.099474 21369 catalog_manager.cc:4841] TS 6c590b9a8f8c4be7a0dee9d3f30eb8fe (127.6.209.194:34839): tablet a133654cf3d74149949f8d8a30ceabe5 (table test-table [id=b14581510955423aa0f92f023e05a6c0]) successfully deleted
I20260926 15:18:08.179299 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 21420
W20260926 15:18:08.196149 21368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.209.193:43579 (error 108)
I20260926 15:18:08.196453 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 21572
I20260926 15:18:08.197999 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:08.197546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:43579 (local address 127.0.0.1:45794)
0926 15:18:08.197686 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:08.197817 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:43579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 15:18:08.198237 21368 catalog_manager.cc:4584] TS 5161fb9c84864bd4bf98f2ce2e724bfa (127.6.209.193:43579): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a133654cf3d74149949f8d8a30ceabe5: Network error: Client connection negotiation failed: client connection to 127.6.209.193:43579: connect: Connection refused (error 111)
I20260926 15:18:08.208570 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:08.197256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39606 (local address 127.6.209.194:34839)
0926 15:18:08.197425 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:08.197433 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:08.197452 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:18:08.197548 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:08.197553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:08.197647 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:08.197737 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:08.199134 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:08.199795 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:08.201101 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:08.201455 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:08.201541 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:08.203843 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:08.203853 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:08.203865 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:08.203894 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:08.206546 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:08.207055 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:08.207060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:08.207062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:08.207166 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:08.207448 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:08.207451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:08.207452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:08.207725 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:08.207923 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:08.208091 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 15:18:08.208264 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:18:08.209206 21695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:18:08.251559 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:08.251109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:43579 (local address 127.0.0.1:45804)
0926 15:18:08.251303 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:08.251399 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:43579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 15:18:08.252571 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:08.252239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:43579 (local address 127.0.0.1:45820)
0926 15:18:08.252380 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:08.252468 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:43579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 15:18:08.292328 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 21572
I20260926 15:18:08.310225 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 21703
I20260926 15:18:08.313117 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:08.312639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:43579 (local address 127.0.0.1:45826)
0926 15:18:08.312864 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:08.312958 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:43579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 15:18:08.314143 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:08.313846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:43579 (local address 127.0.0.1:45842)
0926 15:18:08.313982 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:08.314044 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:43579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 15:18:08.322790 21846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:08.310865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35962 (local address 127.6.209.195:36831)
0926 15:18:08.311027 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:08.311031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:08.311046 (+ 15us) server_negotiation.cc:407] Connection header received
0926 15:18:08.311161 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:08.311165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:08.311219 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:08.311298 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:08.312613 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:08.313634 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:08.315141 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:08.315523 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:08.315615 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:08.317797 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:08.317811 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:08.317826 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:08.317853 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:08.320460 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:08.321081 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:08.321087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:08.321089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:08.321217 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:08.321514 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:08.321517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:08.321519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:08.321841 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:08.322049 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:08.322276 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 15:18:08.322446 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 15:18:08.323351 21826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:18:08.394517 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:08.394149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:43579 (local address 127.0.0.1:45858)
0926 15:18:08.394317 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:08.394388 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:43579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 15:18:08.395543 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:08.395072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:43579 (local address 127.0.0.1:45866)
0926 15:18:08.395359 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:08.395440 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:43579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260926 15:18:08.406857 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 21703
I20260926 15:18:08.422878 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 21350
I20260926 15:18:08.424001 21413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:18:08.530743 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 21350
I20260926 15:18:08.550854 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:40089
--webserver_interface=127.6.209.254
--webserver_port=34359
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:40089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:18:08.854329 21879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:18:08.854588 21879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:18:08.854671 21879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:18:08.861368 21879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:18:08.861454 21879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:18:08.861497 21879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:18:08.861552 21879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:18:08.861596 21879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:18:08.870761 21879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:40089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:40089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=34359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:18:08.871758 21879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:18:08.872843 21879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:18:08.883708 21887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:08.883807 21885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:08.884002 21884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:18:08.884598 21879 server_base.cc:1034] running on GCE node
I20260926 15:18:08.885334 21879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:18:08.886438 21879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:18:08.887633 21879 hybrid_clock.cc:648] HybridClock initialized: now 1790435888887574 us; error 52 us; skew 500 ppm
I20260926 15:18:08.887892 21879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:18:08 dist-test-slave-j6q7 krb5kdc[20801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435888, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:18:08.890800 21879 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:18:08.892138 21879 webserver.cc:460] Webserver started at http://127.6.209.254:34359/ using document root <none> and password file <none>
I20260926 15:18:08.892757 21879 fs_manager.cc:359] Metadata directory not provided
I20260926 15:18:08.892925 21879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:18:08.897742 21879 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:18:08.900198 21894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:08.901453 21879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:18:08.901623 21879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "884244a56c274be9815711313673214a"
format_stamp: "Formatted at 2026-09-26 15:18:03 on dist-test-slave-j6q7"
server_key: "2d6a3b7f7f83efcdb789aed74a14d82c"
server_key_iv: "b211b9142906a90e5312f60a226e1347"
server_key_version: "encryptionkey@0"
I20260926 15:18:08.902629 21879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:18:08.913792 21879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:18:08.917675 21879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:18:08.917919 21879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:18:08.934830 21879 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:40089
I20260926 15:18:08.934855 21945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:40089 every 8 connection(s)
I20260926 15:18:08.936003 21879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:18:08.939913 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 21879
I20260926 15:18:08.940898 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:43579
--local_ip_for_outbound_sockets=127.6.209.193
--tserver_master_addrs=127.6.209.254:40089
--webserver_port=38465
--webserver_interface=127.6.209.193
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:18:08.943246 21946 sys_catalog.cc:263] Verifying existing consensus state
I20260926 15:18:08.947647 21946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a: Bootstrap starting.
I20260926 15:18:08.966579 21946 log.cc:824] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a: Log is configured to *not* fsync() on all Append() calls
I20260926 15:18:08.981123 21946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:18:08.981629 21946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a: Bootstrap complete.
I20260926 15:18:08.990162 21946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884244a56c274be9815711313673214a" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40089 } }
I20260926 15:18:08.990967 21946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 884244a56c274be9815711313673214a, State: Initialized, Role: FOLLOWER
I20260926 15:18:08.991581 21946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884244a56c274be9815711313673214a" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40089 } }
I20260926 15:18:08.991827 21946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:18:08.991961 21946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:18:08.992148 21946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 2 FOLLOWER]: Advancing to term 3
I20260926 15:18:08.995523 21946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884244a56c274be9815711313673214a" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40089 } }
I20260926 15:18:08.995960 21946 leader_election.cc:302] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 884244a56c274be9815711313673214a; no voters:
I20260926 15:18:08.996556 21946 leader_election.cc:288] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 15:18:08.996697 21950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 3 FOLLOWER]: Leader election won for term 3
I20260926 15:18:08.997969 21950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [term 3 LEADER]: Becoming Leader. State: Replica: 884244a56c274be9815711313673214a, State: Running, Role: LEADER
I20260926 15:18:08.998497 21950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884244a56c274be9815711313673214a" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40089 } }
I20260926 15:18:08.998728 21946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:18:09.000825 21952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 884244a56c274be9815711313673214a. Latest consensus state: current_term: 3 leader_uuid: "884244a56c274be9815711313673214a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884244a56c274be9815711313673214a" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40089 } } }
I20260926 15:18:09.001180 21952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [sys.catalog]: This master's current role is: LEADER
I20260926 15:18:09.001690 21951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "884244a56c274be9815711313673214a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "884244a56c274be9815711313673214a" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40089 } } }
I20260926 15:18:09.001963 21951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 884244a56c274be9815711313673214a [sys.catalog]: This master's current role is: LEADER
I20260926 15:18:09.003695 21957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:18:09.011123 21957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:18:09.012533 21957 catalog_manager.cc:1259] Loaded cluster ID: aeea6295143a4ae388e994bdb41048d1
I20260926 15:18:09.012665 21957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 15:18:09.014504 21957 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4d638ce9ad google::LogMessage::Fail() at ??:0
@ 0x7f4d638d188a google::LogMessage::SendToLog() at ??:0
@ 0x7f4d638ce4de google::LogMessage::Flush() at ??:0
@ 0x7f4d638cfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4d79301512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f4d792fe976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4d79386610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f4d79386439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f4d6651a345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4d6651d91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f4d6651d799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f4d664f8eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4d697d26db start_thread at ??:0
@ 0x7f4d6276861f clone at ??:0
W20260926 15:18:09.298305 21948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:18:09.298550 21948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:18:09.298707 21948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:18:09.305569 21948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:18:09.305652 21948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:18:09.305830 21948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:18:09.314494 21948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:43579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=38465
--tserver_master_addrs=127.6.209.254:40089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:18:09.315595 21948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:18:09.316727 21948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:18:09.328819 21985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:09.328878 21987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:09.329228 21984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:18:09.329699 21948 server_base.cc:1034] running on GCE node
I20260926 15:18:09.330201 21948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:18:09.330928 21948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:18:09.332118 21948 hybrid_clock.cc:648] HybridClock initialized: now 1790435889332085 us; error 20 us; skew 500 ppm
I20260926 15:18:09.332365 21948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:18:09 dist-test-slave-j6q7 krb5kdc[20801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435889, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:18:09.335187 21948 init.cc:377] Logged in from keytab as kudu/127.6.209.193@KRBTEST.COM (short username kudu)
I20260926 15:18:09.336426 21948 webserver.cc:460] Webserver started at http://127.6.209.193:38465/ using document root <none> and password file <none>
I20260926 15:18:09.337139 21948 fs_manager.cc:359] Metadata directory not provided
I20260926 15:18:09.337273 21948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:18:09.342061 21948 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 15:18:09.344764 21994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:09.346009 21948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:18:09.346168 21948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "5161fb9c84864bd4bf98f2ce2e724bfa"
format_stamp: "Formatted at 2026-09-26 15:18:04 on dist-test-slave-j6q7"
server_key: "79d81f1983a94a17bc9f9518b67a8f55"
server_key_iv: "66bef5cd196840de3154e2e823da9791"
server_key_version: "encryptionkey@0"
I20260926 15:18:09.347090 21948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:18:09.373708 21948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:18:09.377403 21948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:18:09.377684 21948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:18:09.378638 21948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:18:09.380340 22001 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260926 15:18:09.385442 22001 ts_tablet_manager.cc:1852] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 15:18:09.386353 22001 ts_tablet_manager.cc:1905] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:18:09.393038 22001 ts_tablet_manager.cc:1918] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:18:09.393221 22001 log.cc:1196] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/wal/wals/a133654cf3d74149949f8d8a30ceabe5
I20260926 15:18:09.393723 22001 ts_tablet_manager.cc:1939] T a133654cf3d74149949f8d8a30ceabe5 P 5161fb9c84864bd4bf98f2ce2e724bfa: Deleting consensus metadata
I20260926 15:18:09.394086 21948 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 15:18:09.394181 21948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.000s
I20260926 15:18:09.394264 21948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:18:09.394317 21948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:09.425485 21948 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:43579
I20260926 15:18:09.425590 22107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:43579 every 8 connection(s)
I20260926 15:18:09.426846 21948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:18:09.431373 22110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:09.429710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.193:39795)
0926 15:18:09.430406 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:09.430603 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":297,"threads_started":1}
W20260926 15:18:09.431938 22108 heartbeater.cc:644] Failed to heartbeat to 127.6.209.254:40089 (0 consecutive failures): Network error: Failed to ping master at 127.6.209.254:40089: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
I20260926 15:18:09.432663 22110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:09.432396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.193:41075)
0926 15:18:09.432514 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:09.432575 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 15:18:09.433538 22110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:09.433254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.193:40215)
0926 15:18:09.433360 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:09.433426 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 15:18:09.433807 22108 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 15:18:09.434332 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 21948
I20260926 15:18:09.434887 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:34839
--local_ip_for_outbound_sockets=127.6.209.194
--tserver_master_addrs=127.6.209.254:40089
--webserver_port=33857
--webserver_interface=127.6.209.194
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:18:09.726905 22111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:18:09.727109 22111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:18:09.727182 22111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:18:09.733255 22111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:18:09.733355 22111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:18:09.733502 22111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:18:09.741611 22111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:34839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=33857
--tserver_master_addrs=127.6.209.254:40089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:18:09.742645 22111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:18:09.743731 22111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:18:09.755460 22116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:09.755398 22119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:09.755432 22117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:18:09.756160 22111 server_base.cc:1034] running on GCE node
I20260926 15:18:09.756629 22111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:18:09.757400 22111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:18:09.758574 22111 hybrid_clock.cc:648] HybridClock initialized: now 1790435889758538 us; error 34 us; skew 500 ppm
I20260926 15:18:09.758862 22111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:18:09 dist-test-slave-j6q7 krb5kdc[20801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435889, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:18:09.761694 22111 init.cc:377] Logged in from keytab as kudu/127.6.209.194@KRBTEST.COM (short username kudu)
I20260926 15:18:09.762980 22111 webserver.cc:460] Webserver started at http://127.6.209.194:33857/ using document root <none> and password file <none>
I20260926 15:18:09.763551 22111 fs_manager.cc:359] Metadata directory not provided
I20260926 15:18:09.763676 22111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:18:09.768030 22111 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:18:09.770403 22126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:09.771414 22111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 15:18:09.771560 22111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "6c590b9a8f8c4be7a0dee9d3f30eb8fe"
format_stamp: "Formatted at 2026-09-26 15:18:04 on dist-test-slave-j6q7"
server_key: "7a8dc3e8c6ae79ba4032bc55343ce88b"
server_key_iv: "b5fc3a54d08c31557fa0a1bda370fd88"
server_key_version: "encryptionkey@0"
I20260926 15:18:09.772442 22111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:18:09.798313 22111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:18:09.802026 22111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:18:09.802278 22111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:18:09.803114 22111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:18:09.804198 22111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:18:09.804327 22111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:09.804431 22111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:18:09.804518 22111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:09.834168 22111 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:34839
I20260926 15:18:09.834252 22238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:34839 every 8 connection(s)
I20260926 15:18:09.835465 22111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:18:09.836102 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 22111
I20260926 15:18:09.836874 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:36831
--local_ip_for_outbound_sockets=127.6.209.195
--tserver_master_addrs=127.6.209.254:40089
--webserver_port=36133
--webserver_interface=127.6.209.195
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:18:09.840312 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:09.838656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.194:50057)
0926 15:18:09.839338 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:09.839571 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":297,"threads_started":1}
W20260926 15:18:09.840993 22239 heartbeater.cc:644] Failed to heartbeat to 127.6.209.254:40089 (0 consecutive failures): Network error: Failed to ping master at 127.6.209.254:40089: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
I20260926 15:18:09.841758 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:09.841501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.194:55121)
0926 15:18:09.841618 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:09.841673 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 15:18:09.842576 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:09.842262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.194:55559)
0926 15:18:09.842392 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:09.842474 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 15:18:09.842765 22239 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 15:18:10.120847 22242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:18:10.121083 22242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:18:10.121290 22242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:18:10.127692 22242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:18:10.127770 22242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:18:10.127888 22242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:18:10.136116 22242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:36831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=36133
--tserver_master_addrs=127.6.209.254:40089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:18:10.137192 22242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:18:10.138283 22242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:18:10.149811 22247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:10.149827 22248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:10.149808 22250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:18:10.150333 22242 server_base.cc:1034] running on GCE node
I20260926 15:18:10.151129 22242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:18:10.152004 22242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:18:10.153213 22242 hybrid_clock.cc:648] HybridClock initialized: now 1790435890153172 us; error 56 us; skew 500 ppm
I20260926 15:18:10.153482 22242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:18:10 dist-test-slave-j6q7 krb5kdc[20801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435890, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:18:10.156278 22242 init.cc:377] Logged in from keytab as kudu/127.6.209.195@KRBTEST.COM (short username kudu)
I20260926 15:18:10.157442 22242 webserver.cc:460] Webserver started at http://127.6.209.195:36133/ using document root <none> and password file <none>
I20260926 15:18:10.158004 22242 fs_manager.cc:359] Metadata directory not provided
I20260926 15:18:10.158138 22242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:18:10.162806 22242 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:18:10.165025 22257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:10.166123 22242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:18:10.166282 22242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "c9ea8357080a4741a2d9e555265ccbe9"
format_stamp: "Formatted at 2026-09-26 15:18:05 on dist-test-slave-j6q7"
server_key: "05960ef35ec7dfe6e78a3e2bf9a5c885"
server_key_iv: "d41a0ce4f35f8a07ee2a75186c4739b2"
server_key_version: "encryptionkey@0"
I20260926 15:18:10.167060 22242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:18:10.194332 22242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:18:10.198210 22242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:18:10.198432 22242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:18:10.199244 22242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:18:10.200374 22242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:18:10.200503 22242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:10.200616 22242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:18:10.200698 22242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:10.232393 22242 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:36831
I20260926 15:18:10.232481 22369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:36831 every 8 connection(s)
I20260926 15:18:10.233790 22242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:18:10.236708 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 22242
I20260926 15:18:10.238521 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:10.236320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.195:55117)
0926 15:18:10.237149 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:10.237477 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":258,"threads_started":1}
W20260926 15:18:10.239100 22370 heartbeater.cc:644] Failed to heartbeat to 127.6.209.254:40089 (0 consecutive failures): Network error: Failed to ping master at 127.6.209.254:40089: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
I20260926 15:18:10.239756 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:10.239502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.195:39021)
0926 15:18:10.239612 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:10.239668 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 15:18:10.240525 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:10.240216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.195:35475)
0926 15:18:10.240347 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:10.240422 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 15:18:10.240738 22370 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 15:18:10.434957 22374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:10.434343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.193:57669)
0926 15:18:10.434691 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:10.434771 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":191,"threads_started":1}
I20260926 15:18:10.843931 22375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:10.843250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.194:35905)
0926 15:18:10.843634 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:10.843709 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":181,"threads_started":1}
I20260926 15:18:11.241914 22376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:11.241196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.195:38553)
0926 15:18:11.241628 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:11.241707 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":225,"threads_started":1}
I20260926 15:18:11.436437 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:11.435847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.193:40873)
0926 15:18:11.436223 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:11.436308 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":210,"threads_started":1}
I20260926 15:18:11.845404 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:11.844752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.194:44617)
0926 15:18:11.845192 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:11.845263 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":234,"threads_started":1}
I20260926 15:18:12.243531 22379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:12.242870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.195:53081)
0926 15:18:12.243328 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:12.243394 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":228,"threads_started":1}
I20260926 15:18:12.437988 22380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:12.437278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.193:33369)
0926 15:18:12.437721 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:12.437825 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":203,"threads_started":1}
I20260926 15:18:12.846719 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:12.846120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.194:41851)
0926 15:18:12.846506 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:12.846576 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":190,"threads_started":1}
I20260926 15:18:13.244998 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:13.244307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.195:34951)
0926 15:18:13.244738 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:13.244804 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":268,"threads_started":1}
I20260926 15:18:13.439316 22383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:13.438744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.193:41955)
0926 15:18:13.439109 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:13.439175 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":191,"threads_started":1}
I20260926 15:18:13.848309 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:13.847658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.194:59357)
0926 15:18:13.848077 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:13.848151 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":201,"threads_started":1}
I20260926 15:18:14.246631 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:14.245896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.195:38639)
0926 15:18:14.246382 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:14.246463 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":284,"threads_started":1}
I20260926 15:18:14.440910 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:14.440239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.193:57407)
0926 15:18:14.440654 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:14.440733 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":204,"threads_started":1}
I20260926 15:18:14.849879 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:14.849223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.194:33865)
0926 15:18:14.849685 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:14.849749 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":272,"threads_started":1}
I20260926 15:18:15.248056 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:15.247511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.195:41517)
0926 15:18:15.247858 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:15.247923 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":184,"threads_started":1}
I20260926 15:18:15.442171 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:15.441625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.193:56745)
0926 15:18:15.441979 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:15.442048 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":175,"threads_started":1}
I20260926 15:18:15.851243 22390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:15.850700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.194:54901)
0926 15:18:15.851045 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:15.851117 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":174,"threads_started":1}
I20260926 15:18:16.249475 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:16.248859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.195:35179)
0926 15:18:16.249260 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:16.249338 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":212,"threads_started":1}
I20260926 15:18:16.443710 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:16.442981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.193:34321)
0926 15:18:16.443419 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:16.443513 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":206,"threads_started":1}
I20260926 15:18:16.852651 22393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:16.851952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.194:50729)
0926 15:18:16.852367 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:16.852449 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":188,"threads_started":1}
I20260926 15:18:17.250895 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:17.250246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.195:52583)
0926 15:18:17.250642 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:17.250715 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":172,"threads_started":1}
I20260926 15:18:17.445169 22395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:17.444565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.193:40811)
0926 15:18:17.444969 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:17.445035 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":202,"threads_started":1}
I20260926 15:18:17.854020 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:17.853401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.194:60625)
0926 15:18:17.853794 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:17.853886 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":195,"threads_started":1}
I20260926 15:18:18.252326 22397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:18.251738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.195:48385)
0926 15:18:18.252119 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:18.252189 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":204,"threads_started":1}
I20260926 15:18:18.446416 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:18.445823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.193:41985)
0926 15:18:18.446174 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:18.446251 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":174,"threads_started":1}
I20260926 15:18:18.855715 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:18.854817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.194:46827)
0926 15:18:18.855362 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:18.855513 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":280,"threads_started":1}
I20260926 15:18:19.253880 22400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:19.253117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.195:33871)
0926 15:18:19.253592 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:19.253725 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":204,"threads_started":1}
I20260926 15:18:19.448153 22401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:19.447441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.193:43417)
0926 15:18:19.447918 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:19.447990 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":276,"threads_started":1}
I20260926 15:18:19.857484 22402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:19.856670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.194:36199)
0926 15:18:19.857254 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:19.857331 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":391,"threads_started":1}
I20260926 15:18:20.255278 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:20.254751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.195:47867)
0926 15:18:20.255082 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:20.255146 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":172,"threads_started":1}
I20260926 15:18:20.449604 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:20.449046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.193:53721)
0926 15:18:20.449395 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:20.449461 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":192,"threads_started":1}
I20260926 15:18:20.858803 22405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:20.858221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.194:43287)
0926 15:18:20.858572 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:20.858648 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":182,"threads_started":1}
I20260926 15:18:21.256592 22406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:21.256006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.195:38399)
0926 15:18:21.256358 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:21.256440 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":176,"threads_started":1}
I20260926 15:18:21.450963 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:21.450347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.193:46349)
0926 15:18:21.450701 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:21.450769 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":185,"threads_started":1}
I20260926 15:18:21.860389 22408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:21.859791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.194:50675)
0926 15:18:21.860186 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:21.860258 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":203,"threads_started":1}
I20260926 15:18:22.258035 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:22.257477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.195:55593)
0926 15:18:22.257840 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:22.257907 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":193,"threads_started":1}
I20260926 15:18:22.452257 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:22.451649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.193:42841)
0926 15:18:22.452019 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:22.452101 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":179,"threads_started":1}
I20260926 15:18:22.861824 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:22.861154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.194:55871)
0926 15:18:22.861593 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:22.861677 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":235,"threads_started":1}
I20260926 15:18:23.259582 22412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:23.258870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.195:40049)
0926 15:18:23.259354 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:23.259434 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":238,"threads_started":1}
I20260926 15:18:23.453747 22413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:23.453010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.193:53691)
0926 15:18:23.453502 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:23.453579 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":274,"threads_started":1}
I20260926 15:18:23.863171 22414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:23.862580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.194:42703)
0926 15:18:23.862958 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:23.863027 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":187,"threads_started":1}
I20260926 15:18:24.261175 22415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:24.260388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.195:42847)
0926 15:18:24.260816 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:24.260946 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":219,"threads_started":1}
I20260926 15:18:24.455327 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:24.454735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.193:55219)
0926 15:18:24.455096 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:24.455185 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":180,"threads_started":1}
I20260926 15:18:24.864781 22417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:24.864114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.194:35099)
0926 15:18:24.864534 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:24.864625 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":209,"threads_started":1}
I20260926 15:18:25.249018 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 21948
I20260926 15:18:25.262992 22419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:25.262249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.195:54989)
0926 15:18:25.262719 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:25.262807 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":260,"threads_started":1}
I20260926 15:18:25.457186 22420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:25.456270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.193:60059)
0926 15:18:25.456890 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:25.457002 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":322,"threads_started":1}
I20260926 15:18:25.866428 22421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:25.865826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.194:57777)
0926 15:18:25.866234 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:25.866309 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":190,"threads_started":1}
I20260926 15:18:26.066083 22418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:25.250183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39734 (local address 127.6.209.193:43579)
0926 15:18:25.250887 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:25.250909 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:26.045383 (+794474us) server_negotiation.cc:407] Connection header received
0926 15:18:26.047045 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:26.047056 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:26.047508 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:26.047802 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:26.049131 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:26.050417 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:26.051626 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:26.052268 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:26.052394 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:26.060639 (+ 8245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:26.060645 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:26.060660 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:26.060707 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:26.063270 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:26.063873 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:26.063879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:26.063885 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:26.064052 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:26.064353 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:26.064356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:26.064358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:26.064804 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:26.065105 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:26.065407 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 15:18:26.065788 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":376,"threads_started":1}
I20260926 15:18:26.067271 22103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:18:26.147461 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 21948
I20260926 15:18:26.163766 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 22111
I20260926 15:18:26.179423 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:26.164707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52574 (local address 127.6.209.194:34839)
0926 15:18:26.165213 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:26.165225 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:26.165262 (+ 37us) server_negotiation.cc:407] Connection header received
0926 15:18:26.166707 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:26.166715 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:26.167119 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:26.167426 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:26.168684 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:26.169849 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:26.171014 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:26.171611 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:26.171788 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:26.174086 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:26.174093 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:26.174106 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:26.174138 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:26.176644 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:26.177301 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:26.177314 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:26.177319 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:26.177452 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:26.177776 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:26.177780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:26.177783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:26.178233 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:26.178513 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:26.178834 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 15:18:26.179169 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":252,"threads_started":1}
I20260926 15:18:26.180583 22234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:18:26.256623 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 22111
I20260926 15:18:26.264611 22425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:26.263955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40089 (local address 127.6.209.195:59991)
0926 15:18:26.264387 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:26.264470 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":207,"threads_started":1}
I20260926 15:18:26.272615 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 22242
I20260926 15:18:26.287801 22426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:26.273408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50566 (local address 127.6.209.195:36831)
0926 15:18:26.273925 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:26.273937 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:26.273976 (+ 39us) server_negotiation.cc:407] Connection header received
0926 15:18:26.275459 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:26.275467 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:26.275848 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:26.276099 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:26.277451 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:26.278480 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:26.279651 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:26.280293 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:26.280421 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:26.282709 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:26.282716 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:26.282728 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:26.282755 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:26.285323 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:26.285853 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:26.285858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:26.285863 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:26.285994 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:26.286252 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:26.286255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:26.286257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:26.286734 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:26.286939 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:26.287208 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 15:18:26.287516 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":246,"threads_started":1}
I20260926 15:18:26.289047 22365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:18:26.361752 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 22242
2026-09-26T15:18:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24186 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:18:26 dist-test-slave-j6q7 krb5kdc[22430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:18:26 dist-test-slave-j6q7 krb5kdc[22430](info): set up 2 sockets
Sep 26 15:18:26 dist-test-slave-j6q7 krb5kdc[22430](info): commencing operation
krb5kdc: starting...
W20260926 15:18:27.361176 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:18:27 dist-test-slave-j6q7 krb5kdc[22430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:18:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:18:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:18:27.545989 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44575
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:44575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:18:27.835608 22443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:18:27.835827 22443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:18:27.835902 22443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:18:27.842386 22443 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 15:18:27.842470 22443 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 15:18:27.842501 22443 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 15:18:27.842540 22443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:18:27.842628 22443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:18:27.842722 22443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:18:27.842792 22443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:18:27.842867 22443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:18:27.851549 22443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:44575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:18:27.852566 22443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:18:27.853744 22443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:18:27.865002 22449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:27.864998 22448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:27.865029 22451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:18:27.865454 22443 server_base.cc:1034] running on GCE node
I20260926 15:18:27.866210 22443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:18:27.867280 22443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:18:27.868500 22443 hybrid_clock.cc:648] HybridClock initialized: now 1790435907868450 us; error 45 us; skew 500 ppm
I20260926 15:18:27.868788 22443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:18:27 dist-test-slave-j6q7 krb5kdc[22430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435907, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:18:27.871464 22443 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:18:27.872710 22443 webserver.cc:460] Webserver started at http://127.6.209.254:42513/ using document root <none> and password file <none>
I20260926 15:18:27.873311 22443 fs_manager.cc:359] Metadata directory not provided
I20260926 15:18:27.873435 22443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:18:27.873759 22443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:18:27.876351 22443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c"
format_stamp: "Formatted at 2026-09-26 15:18:27 on dist-test-slave-j6q7"
server_key: "122b0d5561cf4baa693fcf0f9667b395"
server_key_iv: "32846de6a27f06857ea5e47a5488288a"
server_key_version: "encryptionkey@0"
I20260926 15:18:27.877143 22443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c"
format_stamp: "Formatted at 2026-09-26 15:18:27 on dist-test-slave-j6q7"
server_key: "122b0d5561cf4baa693fcf0f9667b395"
server_key_iv: "32846de6a27f06857ea5e47a5488288a"
server_key_version: "encryptionkey@0"
I20260926 15:18:27.881584 22443 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:18:27.884685 22458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:27.885874 22443 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:18:27.886054 22443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c"
format_stamp: "Formatted at 2026-09-26 15:18:27 on dist-test-slave-j6q7"
server_key: "122b0d5561cf4baa693fcf0f9667b395"
server_key_iv: "32846de6a27f06857ea5e47a5488288a"
server_key_version: "encryptionkey@0"
I20260926 15:18:27.886241 22443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:18:27.900411 22443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:18:27.904127 22443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:18:27.904368 22443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:18:27.921733 22443 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:44575
I20260926 15:18:27.921751 22509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:44575 every 8 connection(s)
I20260926 15:18:27.922899 22443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:18:27.926256 22510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:18:27.932780 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 22443
I20260926 15:18:27.932982 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:18:27.933676 6983 external_mini_cluster.cc:1428] Setting key 3801277f4be561804315e525bc4d99bf
I20260926 15:18:27.936091 22510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c: Bootstrap starting.
I20260926 15:18:27.938824 22510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c: Neither blocks nor log segments found. Creating new log.
I20260926 15:18:27.939739 22510 log.cc:824] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c: Log is configured to *not* fsync() on all Append() calls
I20260926 15:18:27.942220 22510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c: No bootstrap required, opened a new log
Sep 26 15:18:27 dist-test-slave-j6q7 krb5kdc[22430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:18:27.949749 22510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44575 } }
I20260926 15:18:27.950043 22510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:18:27.950173 22510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8b515c2a034cbdb9c1d05bdd38c55c, State: Initialized, Role: FOLLOWER
I20260926 15:18:27.950698 22510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44575 } }
I20260926 15:18:27.950950 22510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:18:27.951071 22510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:18:27.951227 22510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:18:27.953397 22510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44575 } }
I20260926 15:18:27.953876 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:27.935539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54176 (local address 127.6.209.254:44575)
0926 15:18:27.936242 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:27.936252 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:27.936287 (+ 35us) server_negotiation.cc:407] Connection header received
0926 15:18:27.937821 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:27.937839 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:27.938127 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:27.938442 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:27.939880 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:27.941231 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:27.942575 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:27.943162 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:27.943334 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:27.947435 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:27.947444 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:27.947457 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:27.947490 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:27.949962 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:27.950679 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:27.950692 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:27.950697 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:27.950837 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:27.951280 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:27.951283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:27.951285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:27.951772 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:27.952083 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:27.952496 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 15:18:27.952915 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":307,"threads_started":1}
I20260926 15:18:27.953872 22510 leader_election.cc:302] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f8b515c2a034cbdb9c1d05bdd38c55c; no voters:
I20260926 15:18:27.954577 22510 leader_election.cc:288] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:18:27.954669 22515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:18:27.955818 22515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 1 LEADER]: Becoming Leader. State: Replica: 5f8b515c2a034cbdb9c1d05bdd38c55c, State: Running, Role: LEADER
I20260926 15:18:27.956362 22515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44575 } }
I20260926 15:18:27.956710 22510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:18:27.959303 22516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f8b515c2a034cbdb9c1d05bdd38c55c. Latest consensus state: current_term: 1 leader_uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44575 } } }
I20260926 15:18:27.959730 22516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [sys.catalog]: This master's current role is: LEADER
I20260926 15:18:27.960533 22517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44575 } } }
I20260926 15:18:27.961056 22517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [sys.catalog]: This master's current role is: LEADER
I20260926 15:18:27.961428 22523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:18:27.966256 22523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:18:27.973165 22523 catalog_manager.cc:1347] Generated new cluster ID: 507c68b71ccf4dc2bd7ec78c4a97dab0
I20260926 15:18:27.973253 22523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:18:27.988732 22523 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:18:27.990278 22523 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:18:27.999176 22523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c: Generated new TSK 0
I20260926 15:18:28.000074 22523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
I20260926 15:18:28.050113 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44575
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:18:28.346310 22536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:18:28.346524 22536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:18:28.346601 22536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:18:28.352442 22536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:18:28.352516 22536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:18:28.352636 22536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:18:28.360975 22536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:18:28.362087 22536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:18:28.363219 22536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:18:28.374629 22542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:28.374668 22541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:28.374593 22544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:18:28.375248 22536 server_base.cc:1034] running on GCE node
I20260926 15:18:28.375763 22536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:18:28.376572 22536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:18:28.377765 22536 hybrid_clock.cc:648] HybridClock initialized: now 1790435908377726 us; error 64 us; skew 500 ppm
I20260926 15:18:28.378059 22536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:18:28 dist-test-slave-j6q7 krb5kdc[22430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435908, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:18:28.380733 22536 init.cc:377] Logged in from keytab as kudu/127.6.209.193@KRBTEST.COM (short username kudu)
I20260926 15:18:28.382038 22536 webserver.cc:460] Webserver started at http://127.6.209.193:44863/ using document root <none> and password file <none>
I20260926 15:18:28.382622 22536 fs_manager.cc:359] Metadata directory not provided
I20260926 15:18:28.382743 22536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:18:28.383020 22536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:18:28.385677 22536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "bab30ea8c2ee4e2fa84ed656a989b777"
format_stamp: "Formatted at 2026-09-26 15:18:28 on dist-test-slave-j6q7"
server_key: "23a13246a18d746cf5642bf2b2c8c0c1"
server_key_iv: "1b5fde8d764dc8eebf53b26fd80d3490"
server_key_version: "encryptionkey@0"
I20260926 15:18:28.386361 22536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "bab30ea8c2ee4e2fa84ed656a989b777"
format_stamp: "Formatted at 2026-09-26 15:18:28 on dist-test-slave-j6q7"
server_key: "23a13246a18d746cf5642bf2b2c8c0c1"
server_key_iv: "1b5fde8d764dc8eebf53b26fd80d3490"
server_key_version: "encryptionkey@0"
I20260926 15:18:28.390848 22536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 15:18:28.393726 22551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:28.394941 22536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260926 15:18:28.395099 22536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "bab30ea8c2ee4e2fa84ed656a989b777"
format_stamp: "Formatted at 2026-09-26 15:18:28 on dist-test-slave-j6q7"
server_key: "23a13246a18d746cf5642bf2b2c8c0c1"
server_key_iv: "1b5fde8d764dc8eebf53b26fd80d3490"
server_key_version: "encryptionkey@0"
I20260926 15:18:28.395273 22536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:18:28.418619 22536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:18:28.421908 22536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:18:28.422143 22536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:18:28.422887 22536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:18:28.423959 22536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:18:28.424057 22536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:28.424198 22536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:18:28.424269 22536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:28.453660 22536 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:46439
I20260926 15:18:28.453723 22663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:46439 every 8 connection(s)
I20260926 15:18:28.454788 22536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:18:28.457219 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 22536
I20260926 15:18:28.457396 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:18:28.458119 6983 external_mini_cluster.cc:1428] Setting key 098b186c8ba75e46df4e01d898e2eaeb
Sep 26 15:18:28 dist-test-slave-j6q7 krb5kdc[22430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435908, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:18:28.474113 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:28.456805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:45829 (local address 127.6.209.254:44575)
0926 15:18:28.457275 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:28.457295 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:28.458081 (+ 786us) server_negotiation.cc:407] Connection header received
0926 15:18:28.459882 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:28.459886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:28.459968 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:28.460126 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:28.462050 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:28.462677 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:28.464984 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:28.465336 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:28.465421 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:28.469097 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:28.469102 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:28.469105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:28.469131 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:28.471415 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:28.472157 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:28.472160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:28.472162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:28.472257 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:28.472651 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:28.472654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:28.472655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:28.472935 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:28.473040 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:28.473696 (+ 656us) server_negotiation.cc:299] Negotiation successful
0926 15:18:28.473860 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":257,"threads_started":1}
I20260926 15:18:28.475473 22667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:28.457266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.193:45829)
0926 15:18:28.457937 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:28.457979 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:28.459591 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:28.460368 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:28.460380 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:28.461119 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:18:28.461840 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:28.461865 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:28.462828 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:28.462835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:28.464832 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:28.464858 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:28.465527 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:28.465534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:28.465792 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:28.466345 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:18:28.466372 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:18:28.468899 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:18:28.471537 (+ 2638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:18:28.471546 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:18:28.471560 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:18:28.472058 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:18:28.472345 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:18:28.472351 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:18:28.472357 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:18:28.472558 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:18:28.473043 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:18:28.473052 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:18:28.473385 (+ 333us) client_negotiation.cc:769] Sending connection context
0926 15:18:28.473630 (+ 245us) client_negotiation.cc:240] Negotiation successful
0926 15:18:28.474134 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":263,"threads_started":1}
I20260926 15:18:28.476737 22664 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44575
I20260926 15:18:28.477011 22664 heartbeater.cc:464] Registering TS with master...
I20260926 15:18:28.477602 22664 heartbeater.cc:511] Master 127.6.209.254:44575 requested a full tablet report, sending...
I20260926 15:18:28.479429 22475 ts_manager.cc:194] Registered new tserver with Master: bab30ea8c2ee4e2fa84ed656a989b777 (127.6.209.193:46439)
I20260926 15:18:28.480773 22475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.193@KRBTEST.COM'} at 127.6.209.193:45829
WARNING: no policy specified for kudu/127.6.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
I20260926 15:18:28.506242 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44575
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:18:28.797398 22670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:18:28.797636 22670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:18:28.797766 22670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:18:28.803897 22670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:18:28.803985 22670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:18:28.804085 22670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:18:28.811982 22670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:18:28.812934 22670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:18:28.813969 22670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:18:28.825729 22676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:28.825753 22678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:28.825811 22675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:18:28.826543 22670 server_base.cc:1034] running on GCE node
I20260926 15:18:28.827095 22670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:18:28.827862 22670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:18:28.829048 22670 hybrid_clock.cc:648] HybridClock initialized: now 1790435908829016 us; error 42 us; skew 500 ppm
I20260926 15:18:28.829325 22670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:18:28 dist-test-slave-j6q7 krb5kdc[22430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435908, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:18:28.832033 22670 init.cc:377] Logged in from keytab as kudu/127.6.209.194@KRBTEST.COM (short username kudu)
I20260926 15:18:28.833344 22670 webserver.cc:460] Webserver started at http://127.6.209.194:42025/ using document root <none> and password file <none>
I20260926 15:18:28.833858 22670 fs_manager.cc:359] Metadata directory not provided
I20260926 15:18:28.833952 22670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:18:28.834184 22670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:18:28.836493 22670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "b0f73802c54b46b09e4d723173fc3806"
format_stamp: "Formatted at 2026-09-26 15:18:28 on dist-test-slave-j6q7"
server_key: "5643f667f15ca64af786e09314fd95bd"
server_key_iv: "47359fd8b7326e0f8ce53018dc4cc8f7"
server_key_version: "encryptionkey@0"
I20260926 15:18:28.837111 22670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "b0f73802c54b46b09e4d723173fc3806"
format_stamp: "Formatted at 2026-09-26 15:18:28 on dist-test-slave-j6q7"
server_key: "5643f667f15ca64af786e09314fd95bd"
server_key_iv: "47359fd8b7326e0f8ce53018dc4cc8f7"
server_key_version: "encryptionkey@0"
I20260926 15:18:28.841243 22670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:18:28.844008 22685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:28.845222 22670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:18:28.845386 22670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "b0f73802c54b46b09e4d723173fc3806"
format_stamp: "Formatted at 2026-09-26 15:18:28 on dist-test-slave-j6q7"
server_key: "5643f667f15ca64af786e09314fd95bd"
server_key_iv: "47359fd8b7326e0f8ce53018dc4cc8f7"
server_key_version: "encryptionkey@0"
I20260926 15:18:28.845542 22670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:18:28.865584 22670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:18:28.869167 22670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:18:28.869422 22670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:18:28.870146 22670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:18:28.871160 22670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:18:28.871220 22670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:28.871373 22670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:18:28.871416 22670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:28.900357 22670 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:34287
I20260926 15:18:28.900439 22797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:34287 every 8 connection(s)
I20260926 15:18:28.901603 22670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:18:28.909096 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 22670
I20260926 15:18:28.909271 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:18:28.909668 6983 external_mini_cluster.cc:1428] Setting key 7c69dc4ddb768c60ddaccab93ed7bf97
Sep 26 15:18:28 dist-test-slave-j6q7 krb5kdc[22430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435908, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:18:28.918555 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:28.903775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:37457 (local address 127.6.209.254:44575)
0926 15:18:28.903990 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:28.903997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:28.905188 (+ 1191us) server_negotiation.cc:407] Connection header received
0926 15:18:28.906728 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:28.906732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:28.906794 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:28.906871 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:28.908089 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:28.908707 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:28.910348 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:28.910806 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:28.910921 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:28.913734 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:28.913738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:28.913741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:28.913772 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:28.916034 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:28.916638 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:28.916647 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:28.916649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:28.916762 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:28.917150 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:28.917155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:28.917157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:28.917383 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:28.917518 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:28.918054 (+ 536us) server_negotiation.cc:299] Negotiation successful
0926 15:18:28.918243 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 15:18:28.919371 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:28.904331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.194:37457)
0926 15:18:28.905047 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:28.905090 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:28.906516 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:28.907055 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:28.907062 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:28.907487 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:18:28.907941 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:28.907955 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:28.908865 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:28.908868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:28.910169 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:28.910184 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:28.911002 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:28.911009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:28.911204 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:28.911751 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:18:28.911778 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:18:28.913554 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:18:28.916152 (+ 2598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:18:28.916159 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:18:28.916171 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:18:28.916525 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:18:28.916885 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:18:28.916889 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:18:28.916892 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:18:28.917047 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:18:28.917497 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:18:28.917503 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:18:28.917759 (+ 256us) client_negotiation.cc:769] Sending connection context
0926 15:18:28.917976 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 15:18:28.918381 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":252,"threads_started":1}
I20260926 15:18:28.920934 22798 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44575
I20260926 15:18:28.921311 22798 heartbeater.cc:464] Registering TS with master...
I20260926 15:18:28.921873 22798 heartbeater.cc:511] Master 127.6.209.254:44575 requested a full tablet report, sending...
I20260926 15:18:28.923230 22475 ts_manager.cc:194] Registered new tserver with Master: b0f73802c54b46b09e4d723173fc3806 (127.6.209.194:34287)
I20260926 15:18:28.924058 22475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.194@KRBTEST.COM'} at 127.6.209.194:37457
WARNING: no policy specified for kudu/127.6.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
I20260926 15:18:28.957796 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44575
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:18:29.247051 22803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:18:29.247283 22803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:18:29.247363 22803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:18:29.254009 22803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:18:29.254097 22803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:18:29.254209 22803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:18:29.262646 22803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:44575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:18:29.263602 22803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:18:29.264743 22803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:18:29.276310 22808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:29.276360 22809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:29.276377 22811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:18:29.276883 22803 server_base.cc:1034] running on GCE node
I20260926 15:18:29.277410 22803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:18:29.278198 22803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:18:29.279399 22803 hybrid_clock.cc:648] HybridClock initialized: now 1790435909279364 us; error 46 us; skew 500 ppm
I20260926 15:18:29.279690 22803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:18:29 dist-test-slave-j6q7 krb5kdc[22430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435909, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:18:29.282617 22803 init.cc:377] Logged in from keytab as kudu/127.6.209.195@KRBTEST.COM (short username kudu)
I20260926 15:18:29.283860 22803 webserver.cc:460] Webserver started at http://127.6.209.195:33707/ using document root <none> and password file <none>
I20260926 15:18:29.284471 22803 fs_manager.cc:359] Metadata directory not provided
I20260926 15:18:29.284613 22803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:18:29.284968 22803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:18:29.287452 22803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "85783935ae1944c7afbb471f87c8abef"
format_stamp: "Formatted at 2026-09-26 15:18:29 on dist-test-slave-j6q7"
server_key: "f331207c1de312149c3bf1cfba35dc93"
server_key_iv: "b470db8e3f3ece21b33824a04d96bfc1"
server_key_version: "encryptionkey@0"
I20260926 15:18:29.288110 22803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "85783935ae1944c7afbb471f87c8abef"
format_stamp: "Formatted at 2026-09-26 15:18:29 on dist-test-slave-j6q7"
server_key: "f331207c1de312149c3bf1cfba35dc93"
server_key_iv: "b470db8e3f3ece21b33824a04d96bfc1"
server_key_version: "encryptionkey@0"
I20260926 15:18:29.292510 22803 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260926 15:18:29.295521 22818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:29.296548 22803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:18:29.296702 22803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "85783935ae1944c7afbb471f87c8abef"
format_stamp: "Formatted at 2026-09-26 15:18:29 on dist-test-slave-j6q7"
server_key: "f331207c1de312149c3bf1cfba35dc93"
server_key_iv: "b470db8e3f3ece21b33824a04d96bfc1"
server_key_version: "encryptionkey@0"
I20260926 15:18:29.296895 22803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:18:29.327946 22803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:18:29.331862 22803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:18:29.332106 22803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:18:29.332952 22803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:18:29.334093 22803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:18:29.334196 22803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:29.334354 22803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:18:29.334426 22803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:29.364732 22803 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:37809
I20260926 15:18:29.364791 22930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:37809 every 8 connection(s)
I20260926 15:18:29.365882 22803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:18:29.374853 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 22803
I20260926 15:18:29.375038 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:18:29.375556 6983 external_mini_cluster.cc:1428] Setting key d91b0a5637c9383eb611dbe5901ff6b9
Sep 26 15:18:29 dist-test-slave-j6q7 krb5kdc[22430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435909, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:18:29.383212 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:29.367914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:37331 (local address 127.6.209.254:44575)
0926 15:18:29.368067 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:29.368071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:29.369072 (+ 1001us) server_negotiation.cc:407] Connection header received
0926 15:18:29.370650 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:29.370654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:29.370719 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:29.370800 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:29.372061 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.372670 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.374014 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.374423 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.374514 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:29.377933 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:29.377938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:29.377942 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:29.377996 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:29.380490 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:29.381254 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:29.381259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:29.381263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:29.381382 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:29.381778 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:29.381782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:29.381785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:29.382020 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:29.382156 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:29.382708 (+ 552us) server_negotiation.cc:299] Negotiation successful
0926 15:18:29.382899 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 15:18:29.383991 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:29.368287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.195:37331)
0926 15:18:29.368923 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:29.368971 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:29.370428 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:29.370988 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:29.370995 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:29.371469 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:18:29.371930 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:29.371944 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.372784 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.372787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:29.373894 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:29.373906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.374634 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.374643 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:29.374924 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:29.375527 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:18:29.375559 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:18:29.377753 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:18:29.380620 (+ 2867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:18:29.380628 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:18:29.380641 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:18:29.381121 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:18:29.381483 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:18:29.381489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:18:29.381493 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:18:29.381656 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:18:29.382132 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:18:29.382138 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:18:29.382410 (+ 272us) client_negotiation.cc:769] Sending connection context
0926 15:18:29.382630 (+ 220us) client_negotiation.cc:240] Negotiation successful
0926 15:18:29.383050 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":228,"threads_started":1}
I20260926 15:18:29.385191 22931 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44575
I20260926 15:18:29.385448 22931 heartbeater.cc:464] Registering TS with master...
I20260926 15:18:29.386049 22931 heartbeater.cc:511] Master 127.6.209.254:44575 requested a full tablet report, sending...
I20260926 15:18:29.387168 22475 ts_manager.cc:194] Registered new tserver with Master: 85783935ae1944c7afbb471f87c8abef (127.6.209.195:37809)
I20260926 15:18:29.387836 22475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.195@KRBTEST.COM'} at 127.6.209.195:37331
I20260926 15:18:29.390087 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:18:29.405115 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:29.393984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54188 (local address 127.6.209.254:44575)
0926 15:18:29.394129 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:29.394132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:29.394564 (+ 432us) server_negotiation.cc:407] Connection header received
0926 15:18:29.394676 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:29.394679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:29.394726 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:29.394806 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:29.395997 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.396606 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.397908 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.398246 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.398326 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:29.400334 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:29.400338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:29.400341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:29.400369 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:29.402698 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:29.403270 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:29.403278 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:29.403281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:29.403376 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:29.403689 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:29.403696 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:29.403699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:29.403983 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:29.404219 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:29.404545 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 15:18:29.404745 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 15:18:29.408409 22475 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:18:29.414369 22475 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:18:29.440361 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:29.425268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34514 (local address 127.6.209.195:37809)
0926 15:18:29.426807 (+ 1539us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:29.426818 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:29.427412 (+ 594us) server_negotiation.cc:407] Connection header received
0926 15:18:29.427519 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:29.427529 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:29.427782 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:29.429053 (+ 1271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:18:29.430118 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.433095 (+ 2977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.437775 (+ 4680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.439228 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.439357 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:29.439583 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:29.439776 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 15:18:29.439946 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1360,"thread_start_us":1215,"threads_started":1}
I20260926 15:18:29.440361 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:29.424667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:37809 (local address 127.0.0.1:34514)
0926 15:18:29.427184 (+ 2517us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:29.427199 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:29.427293 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:29.429390 (+ 2097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:29.429413 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:29.429461 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:18:29.429886 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:29.429897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.433311 (+ 3414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.433318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:29.437570 (+ 4252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:29.437589 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.439411 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.439417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:29.439584 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:29.439627 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 15:18:29.439735 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 15:18:29.439906 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2395,"thread_start_us":2283,"threads_started":1}
I20260926 15:18:29.444355 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:29.424449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:34287 (local address 127.0.0.1:51780)
0926 15:18:29.425456 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:29.425505 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:29.425683 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:29.437815 (+ 12132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:29.437820 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:29.437841 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:18:29.438222 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:29.438236 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.440046 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.440053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:29.441898 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:29.441909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.443461 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.443466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:29.443696 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:29.443728 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 15:18:29.443814 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 15:18:29.443964 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":48,"thread_start_us":1353,"threads_started":1}
I20260926 15:18:29.444350 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:29.424632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51780 (local address 127.6.209.194:34287)
0926 15:18:29.437186 (+ 12554us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:29.437214 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:29.437247 (+ 33us) server_negotiation.cc:407] Connection header received
0926 15:18:29.437351 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:29.437361 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:29.437628 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:29.437825 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:18:29.438571 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.439882 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.442045 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.443257 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.443370 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:29.443523 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:29.443881 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 15:18:29.444047 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12357,"thread_start_us":13341,"threads_started":1}
I20260926 15:18:29.444185 22866 tablet_service.cc:1467] Processing CreateTablet for tablet 72a6dc89fc6c40df820e90b5b77f1d4c (DEFAULT_TABLE table=test-table [id=0b65435820d34044b007924cd314e529]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:18:29.445604 22866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a6dc89fc6c40df820e90b5b77f1d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:18:29.446429 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:29.427287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:46439 (local address 127.0.0.1:51462)
0926 15:18:29.438864 (+ 11577us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:29.438880 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:29.438986 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:29.439668 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:29.439674 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:29.439696 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:18:29.440079 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:29.440093 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.442331 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.442335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:29.444090 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:29.444103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.445753 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.445759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:29.445905 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:29.445946 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 15:18:29.446034 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 15:18:29.446134 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11452,"mutex_wait_us":42,"thread_start_us":11337,"threads_started":1}
I20260926 15:18:29.446494 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:29.427561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51462 (local address 127.6.209.193:46439)
0926 15:18:29.438489 (+ 10928us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:29.438496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:29.438983 (+ 487us) server_negotiation.cc:407] Connection header received
0926 15:18:29.439106 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:29.439113 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:29.439317 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:29.439476 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:18:29.440282 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.442202 (+ 1920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.444260 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.445583 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.445716 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:29.445871 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:29.446073 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 15:18:29.446213 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10784,"thread_start_us":10800,"threads_started":1}
I20260926 15:18:29.447817 22733 tablet_service.cc:1467] Processing CreateTablet for tablet 72a6dc89fc6c40df820e90b5b77f1d4c (DEFAULT_TABLE table=test-table [id=0b65435820d34044b007924cd314e529]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:18:29.449000 22733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a6dc89fc6c40df820e90b5b77f1d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:18:29.457269 22599 tablet_service.cc:1467] Processing CreateTablet for tablet 72a6dc89fc6c40df820e90b5b77f1d4c (DEFAULT_TABLE table=test-table [id=0b65435820d34044b007924cd314e529]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:18:29.458439 22599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a6dc89fc6c40df820e90b5b77f1d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:18:29.459723 22949 tablet_bootstrap.cc:492] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef: Bootstrap starting.
I20260926 15:18:29.462996 22949 tablet_bootstrap.cc:654] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef: Neither blocks nor log segments found. Creating new log.
I20260926 15:18:29.464471 22949 log.cc:824] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef: Log is configured to *not* fsync() on all Append() calls
I20260926 15:18:29.465936 22950 tablet_bootstrap.cc:492] T 72a6dc89fc6c40df820e90b5b77f1d4c P b0f73802c54b46b09e4d723173fc3806: Bootstrap starting.
I20260926 15:18:29.467367 22949 tablet_bootstrap.cc:492] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef: No bootstrap required, opened a new log
I20260926 15:18:29.467598 22949 ts_tablet_manager.cc:1397] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260926 15:18:29.468307 22950 tablet_bootstrap.cc:654] T 72a6dc89fc6c40df820e90b5b77f1d4c P b0f73802c54b46b09e4d723173fc3806: Neither blocks nor log segments found. Creating new log.
I20260926 15:18:29.469249 22950 log.cc:824] T 72a6dc89fc6c40df820e90b5b77f1d4c P b0f73802c54b46b09e4d723173fc3806: Log is configured to *not* fsync() on all Append() calls
I20260926 15:18:29.470418 22953 tablet_bootstrap.cc:492] T 72a6dc89fc6c40df820e90b5b77f1d4c P bab30ea8c2ee4e2fa84ed656a989b777: Bootstrap starting.
I20260926 15:18:29.471766 22950 tablet_bootstrap.cc:492] T 72a6dc89fc6c40df820e90b5b77f1d4c P b0f73802c54b46b09e4d723173fc3806: No bootstrap required, opened a new log
I20260926 15:18:29.472009 22950 ts_tablet_manager.cc:1397] T 72a6dc89fc6c40df820e90b5b77f1d4c P b0f73802c54b46b09e4d723173fc3806: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 15:18:29.473428 22953 tablet_bootstrap.cc:654] T 72a6dc89fc6c40df820e90b5b77f1d4c P bab30ea8c2ee4e2fa84ed656a989b777: Neither blocks nor log segments found. Creating new log.
I20260926 15:18:29.474506 22953 log.cc:824] T 72a6dc89fc6c40df820e90b5b77f1d4c P bab30ea8c2ee4e2fa84ed656a989b777: Log is configured to *not* fsync() on all Append() calls
I20260926 15:18:29.476032 22949 raft_consensus.cc:348] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } } peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } }
I20260926 15:18:29.476325 22949 raft_consensus.cc:374] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:18:29.476437 22949 raft_consensus.cc:729] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85783935ae1944c7afbb471f87c8abef, State: Initialized, Role: FOLLOWER
I20260926 15:18:29.477007 22953 tablet_bootstrap.cc:492] T 72a6dc89fc6c40df820e90b5b77f1d4c P bab30ea8c2ee4e2fa84ed656a989b777: No bootstrap required, opened a new log
I20260926 15:18:29.476984 22949 consensus_queue.cc:260] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } } peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } }
I20260926 15:18:29.477216 22953 ts_tablet_manager.cc:1397] T 72a6dc89fc6c40df820e90b5b77f1d4c P bab30ea8c2ee4e2fa84ed656a989b777: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 15:18:29.477906 22931 heartbeater.cc:503] Master 127.6.209.254:44575 was elected leader, sending a full tablet report...
I20260926 15:18:29.478399 22949 ts_tablet_manager.cc:1428] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s
I20260926 15:18:29.481185 22950 raft_consensus.cc:348] T 72a6dc89fc6c40df820e90b5b77f1d4c P b0f73802c54b46b09e4d723173fc3806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } } peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } }
I20260926 15:18:29.481550 22950 raft_consensus.cc:374] T 72a6dc89fc6c40df820e90b5b77f1d4c P b0f73802c54b46b09e4d723173fc3806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:18:29.481662 22950 raft_consensus.cc:729] T 72a6dc89fc6c40df820e90b5b77f1d4c P b0f73802c54b46b09e4d723173fc3806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f73802c54b46b09e4d723173fc3806, State: Initialized, Role: FOLLOWER
I20260926 15:18:29.482153 22950 consensus_queue.cc:260] T 72a6dc89fc6c40df820e90b5b77f1d4c P b0f73802c54b46b09e4d723173fc3806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } } peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } }
I20260926 15:18:29.483151 22798 heartbeater.cc:503] Master 127.6.209.254:44575 was elected leader, sending a full tablet report...
I20260926 15:18:29.483376 22664 heartbeater.cc:503] Master 127.6.209.254:44575 was elected leader, sending a full tablet report...
I20260926 15:18:29.483729 22950 ts_tablet_manager.cc:1428] T 72a6dc89fc6c40df820e90b5b77f1d4c P b0f73802c54b46b09e4d723173fc3806: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260926 15:18:29.487038 22953 raft_consensus.cc:348] T 72a6dc89fc6c40df820e90b5b77f1d4c P bab30ea8c2ee4e2fa84ed656a989b777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } } peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } }
I20260926 15:18:29.487358 22953 raft_consensus.cc:374] T 72a6dc89fc6c40df820e90b5b77f1d4c P bab30ea8c2ee4e2fa84ed656a989b777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:18:29.487438 22953 raft_consensus.cc:729] T 72a6dc89fc6c40df820e90b5b77f1d4c P bab30ea8c2ee4e2fa84ed656a989b777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bab30ea8c2ee4e2fa84ed656a989b777, State: Initialized, Role: FOLLOWER
I20260926 15:18:29.487913 22953 consensus_queue.cc:260] T 72a6dc89fc6c40df820e90b5b77f1d4c P bab30ea8c2ee4e2fa84ed656a989b777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } } peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } }
I20260926 15:18:29.489193 22953 ts_tablet_manager.cc:1428] T 72a6dc89fc6c40df820e90b5b77f1d4c P bab30ea8c2ee4e2fa84ed656a989b777: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260926 15:18:29.570613 22955 raft_consensus.cc:482] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:18:29.570900 22955 raft_consensus.cc:504] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } } peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } }
I20260926 15:18:29.572299 22955 leader_election.cc:288] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0f73802c54b46b09e4d723173fc3806 (127.6.209.194:34287), bab30ea8c2ee4e2fa84ed656a989b777 (127.6.209.193:46439)
I20260926 15:18:29.578184 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:29.572676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:38283 (local address 127.6.209.193:46439)
0926 15:18:29.572886 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:29.572892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:29.573064 (+ 172us) server_negotiation.cc:407] Connection header received
0926 15:18:29.573295 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:29.573300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:29.573369 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:29.573462 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:18:29.573852 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.574629 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.576415 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.577540 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.577632 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:29.577674 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:29.577951 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 15:18:29.578023 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 15:18:29.578277 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:29.572700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:46439 (local address 127.6.209.195:38283)
0926 15:18:29.572967 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:29.572986 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:29.573134 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:29.573503 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:29.573509 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:29.573533 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:18:29.573729 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:29.573741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.574750 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.574756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:29.576268 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:29.576279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.577652 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.577657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:29.577814 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:29.577843 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 15:18:29.577927 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 15:18:29.578037 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":17}
I20260926 15:18:29.579170 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:29.572781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:34287 (local address 127.6.209.195:38131)
0926 15:18:29.573480 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:29.573499 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:29.573651 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:29.574019 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:29.574023 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:29.574044 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:18:29.574339 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:29.574347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.575606 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.575610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:29.577332 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:29.577345 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.578621 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.578629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:29.578777 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:29.578805 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 15:18:29.578877 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 15:18:29.578953 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":40,"thread_start_us":352,"threads_started":1}
I20260926 15:18:29.579258 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:29.572753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:38131 (local address 127.6.209.194:34287)
0926 15:18:29.572951 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:29.572955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:29.573563 (+ 608us) server_negotiation.cc:407] Connection header received
0926 15:18:29.573801 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:29.573809 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:29.573911 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:29.574007 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:18:29.574490 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.575477 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.577467 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.578451 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.578576 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:29.578618 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:29.578936 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 15:18:29.579047 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 15:18:29.579205 22619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6dc89fc6c40df820e90b5b77f1d4c" candidate_uuid: "85783935ae1944c7afbb471f87c8abef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" is_pre_election: true
I20260926 15:18:29.579764 22619 raft_consensus.cc:2454] T 72a6dc89fc6c40df820e90b5b77f1d4c P bab30ea8c2ee4e2fa84ed656a989b777 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85783935ae1944c7afbb471f87c8abef in term 0.
I20260926 15:18:29.580263 22753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6dc89fc6c40df820e90b5b77f1d4c" candidate_uuid: "85783935ae1944c7afbb471f87c8abef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f73802c54b46b09e4d723173fc3806" is_pre_election: true
I20260926 15:18:29.580380 22822 leader_election.cc:302] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85783935ae1944c7afbb471f87c8abef, bab30ea8c2ee4e2fa84ed656a989b777; no voters:
I20260926 15:18:29.580722 22955 raft_consensus.cc:2781] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:18:29.580739 22753 raft_consensus.cc:2454] T 72a6dc89fc6c40df820e90b5b77f1d4c P b0f73802c54b46b09e4d723173fc3806 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85783935ae1944c7afbb471f87c8abef in term 0.
I20260926 15:18:29.580899 22955 raft_consensus.cc:482] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:18:29.581007 22955 raft_consensus.cc:3037] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:18:29.582906 22955 raft_consensus.cc:504] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } } peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } }
I20260926 15:18:29.583410 22955 leader_election.cc:288] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef [CANDIDATE]: Term 1 election: Requested vote from peers b0f73802c54b46b09e4d723173fc3806 (127.6.209.194:34287), bab30ea8c2ee4e2fa84ed656a989b777 (127.6.209.193:46439)
I20260926 15:18:29.583839 22753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6dc89fc6c40df820e90b5b77f1d4c" candidate_uuid: "85783935ae1944c7afbb471f87c8abef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f73802c54b46b09e4d723173fc3806"
I20260926 15:18:29.583843 22619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a6dc89fc6c40df820e90b5b77f1d4c" candidate_uuid: "85783935ae1944c7afbb471f87c8abef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bab30ea8c2ee4e2fa84ed656a989b777"
I20260926 15:18:29.584048 22619 raft_consensus.cc:3037] T 72a6dc89fc6c40df820e90b5b77f1d4c P bab30ea8c2ee4e2fa84ed656a989b777 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:18:29.584048 22753 raft_consensus.cc:3037] T 72a6dc89fc6c40df820e90b5b77f1d4c P b0f73802c54b46b09e4d723173fc3806 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:18:29.587024 22753 raft_consensus.cc:2454] T 72a6dc89fc6c40df820e90b5b77f1d4c P b0f73802c54b46b09e4d723173fc3806 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85783935ae1944c7afbb471f87c8abef in term 1.
I20260926 15:18:29.587024 22619 raft_consensus.cc:2454] T 72a6dc89fc6c40df820e90b5b77f1d4c P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85783935ae1944c7afbb471f87c8abef in term 1.
I20260926 15:18:29.587352 22822 leader_election.cc:302] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85783935ae1944c7afbb471f87c8abef, bab30ea8c2ee4e2fa84ed656a989b777; no voters:
I20260926 15:18:29.587695 22955 raft_consensus.cc:2781] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:18:29.588097 22955 raft_consensus.cc:686] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef [term 1 LEADER]: Becoming Leader. State: Replica: 85783935ae1944c7afbb471f87c8abef, State: Running, Role: LEADER
I20260926 15:18:29.588518 22955 consensus_queue.cc:237] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } } peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } }
I20260926 15:18:29.592790 22474 catalog_manager.cc:5495] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef reported cstate change: term changed from 0 to 1, leader changed from <none> to 85783935ae1944c7afbb471f87c8abef (127.6.209.195). New cstate: current_term: 1 leader_uuid: "85783935ae1944c7afbb471f87c8abef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } health_report { overall_health: UNKNOWN } } }
W20260926 15:18:29.617245 22932 tablet.cc:2369] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:18:29.621778 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:29.617051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34520 (local address 127.6.209.195:37809)
0926 15:18:29.617236 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:29.617243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:29.617310 (+ 67us) server_negotiation.cc:407] Connection header received
0926 15:18:29.617512 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:29.617515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:29.617584 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:29.617712 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:18:29.618036 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.618638 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.619781 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.620184 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.620314 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:29.620774 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:18:29.621236 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:18:29.621389 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:29.621486 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 15:18:29.621553 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 15:18:29.628566 22619 raft_consensus.cc:1261] T 72a6dc89fc6c40df820e90b5b77f1d4c P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Refusing update from remote peer 85783935ae1944c7afbb471f87c8abef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:18:29.628584 22753 raft_consensus.cc:1261] T 72a6dc89fc6c40df820e90b5b77f1d4c P b0f73802c54b46b09e4d723173fc3806 [term 1 FOLLOWER]: Refusing update from remote peer 85783935ae1944c7afbb471f87c8abef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:18:29.629359 22959 consensus_queue.cc:1035] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:18:29.629643 22955 consensus_queue.cc:1035] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef [LEADER]: Connected to new peer: Peer: permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:18:29.639452 22962 mvcc.cc:205] Tried to move back new op lower bound from 7333625485831016448 to 7333625485678133248. Current Snapshot: MvccSnapshot[applied={T|T < 7333625485831016448}]
I20260926 15:18:29.650930 22474 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:18:29.651156 22474 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:18:29.654353 22474 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c: Sending DeleteTablet for 3 replicas of tablet 72a6dc89fc6c40df820e90b5b77f1d4c
I20260926 15:18:29.655344 22733 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a6dc89fc6c40df820e90b5b77f1d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:18:29 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:51780
I20260926 15:18:29.655437 22866 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a6dc89fc6c40df820e90b5b77f1d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:18:29 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:34514
I20260926 15:18:29.655547 22599 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a6dc89fc6c40df820e90b5b77f1d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:18:29 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:51462
I20260926 15:18:29.656466 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 22536
I20260926 15:18:29.671617 22973 tablet_replica.cc:330] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef: stopping tablet replica
I20260926 15:18:29.672142 22973 raft_consensus.cc:2229] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:18:29.673303 22972 tablet_replica.cc:330] T 72a6dc89fc6c40df820e90b5b77f1d4c P b0f73802c54b46b09e4d723173fc3806: stopping tablet replica
I20260926 15:18:29.673767 22972 raft_consensus.cc:2229] T 72a6dc89fc6c40df820e90b5b77f1d4c P b0f73802c54b46b09e4d723173fc3806 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:18:29.674391 22972 raft_consensus.cc:2258] T 72a6dc89fc6c40df820e90b5b77f1d4c P b0f73802c54b46b09e4d723173fc3806 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:18:29.674993 22973 raft_consensus.cc:2258] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:18:29.675071 22974 tablet_replica.cc:330] T 72a6dc89fc6c40df820e90b5b77f1d4c P bab30ea8c2ee4e2fa84ed656a989b777: stopping tablet replica
I20260926 15:18:29.675416 22974 raft_consensus.cc:2229] T 72a6dc89fc6c40df820e90b5b77f1d4c P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:18:29.675935 22974 raft_consensus.cc:2258] T 72a6dc89fc6c40df820e90b5b77f1d4c P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:18:29.677445 22973 ts_tablet_manager.cc:1905] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:18:29.677891 22974 ts_tablet_manager.cc:1905] T 72a6dc89fc6c40df820e90b5b77f1d4c P bab30ea8c2ee4e2fa84ed656a989b777: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:18:29.678946 22972 ts_tablet_manager.cc:1905] T 72a6dc89fc6c40df820e90b5b77f1d4c P b0f73802c54b46b09e4d723173fc3806: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 15:18:29 dist-test-slave-j6q7 krb5kdc[22430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.193@KRBTEST.COM
I20260926 15:18:29.685467 22974 ts_tablet_manager.cc:1918] T 72a6dc89fc6c40df820e90b5b77f1d4c P bab30ea8c2ee4e2fa84ed656a989b777: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:18:29.685716 22974 log.cc:1196] T 72a6dc89fc6c40df820e90b5b77f1d4c P bab30ea8c2ee4e2fa84ed656a989b777: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/wal/wals/72a6dc89fc6c40df820e90b5b77f1d4c
I20260926 15:18:29.686276 22974 ts_tablet_manager.cc:1939] T 72a6dc89fc6c40df820e90b5b77f1d4c P bab30ea8c2ee4e2fa84ed656a989b777: Deleting consensus metadata
I20260926 15:18:29.688014 22462 catalog_manager.cc:4841] TS bab30ea8c2ee4e2fa84ed656a989b777 (127.6.209.193:46439): tablet 72a6dc89fc6c40df820e90b5b77f1d4c (table test-table [id=0b65435820d34044b007924cd314e529]) successfully deleted
I20260926 15:18:29.689877 22972 ts_tablet_manager.cc:1918] T 72a6dc89fc6c40df820e90b5b77f1d4c P b0f73802c54b46b09e4d723173fc3806: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:18:29.690099 22972 log.cc:1196] T 72a6dc89fc6c40df820e90b5b77f1d4c P b0f73802c54b46b09e4d723173fc3806: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/72a6dc89fc6c40df820e90b5b77f1d4c
I20260926 15:18:29.690351 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:29.657684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51478 (local address 127.6.209.193:46439)
0926 15:18:29.657874 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:29.657883 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:29.672395 (+ 14512us) server_negotiation.cc:407] Connection header received
0926 15:18:29.672620 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:29.672627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:29.672726 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:29.672863 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:29.674540 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.675704 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.677610 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.678154 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.678263 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:29.683436 (+ 5173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:29.683447 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:29.683467 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:29.683508 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:29.687217 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:29.687937 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:29.687945 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:29.687947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:29.688083 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:29.688519 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:29.688523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:29.688526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:29.688982 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:29.689231 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:29.689801 (+ 570us) server_negotiation.cc:299] Negotiation successful
0926 15:18:29.690007 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 15:18:29.691040 22973 ts_tablet_manager.cc:1918] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:18:29.691268 22973 log.cc:1196] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/72a6dc89fc6c40df820e90b5b77f1d4c
I20260926 15:18:29.691288 22972 ts_tablet_manager.cc:1939] T 72a6dc89fc6c40df820e90b5b77f1d4c P b0f73802c54b46b09e4d723173fc3806: Deleting consensus metadata
I20260926 15:18:29.691545 22659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:18:29.691820 22973 ts_tablet_manager.cc:1939] T 72a6dc89fc6c40df820e90b5b77f1d4c P 85783935ae1944c7afbb471f87c8abef: Deleting consensus metadata
I20260926 15:18:29.693055 22459 catalog_manager.cc:4841] TS b0f73802c54b46b09e4d723173fc3806 (127.6.209.194:34287): tablet 72a6dc89fc6c40df820e90b5b77f1d4c (table test-table [id=0b65435820d34044b007924cd314e529]) successfully deleted
I20260926 15:18:29.693672 22462 catalog_manager.cc:4841] TS 85783935ae1944c7afbb471f87c8abef (127.6.209.195:37809): tablet 72a6dc89fc6c40df820e90b5b77f1d4c (table test-table [id=0b65435820d34044b007924cd314e529]) successfully deleted
I20260926 15:18:29.795545 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 22536
I20260926 15:18:29.811839 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 22670
Sep 26 15:18:29 dist-test-slave-j6q7 krb5kdc[22430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.194@KRBTEST.COM
I20260926 15:18:29.828078 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:29.812693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51782 (local address 127.6.209.194:34287)
0926 15:18:29.812913 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:29.812919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:29.812943 (+ 24us) server_negotiation.cc:407] Connection header received
0926 15:18:29.813114 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:29.813120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:29.813201 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:29.813317 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:29.814518 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.815679 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.817276 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.817841 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.817969 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:29.822086 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:29.822105 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:29.822139 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:29.822177 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:29.825634 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:29.826245 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:29.826253 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:29.826257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:29.826402 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:29.826713 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:29.826718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:29.826722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:29.827141 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:29.827417 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:29.827559 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 15:18:29.827749 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 15:18:29.828670 22793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:18:29.909438 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 22670
I20260926 15:18:29.930361 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 22803
Sep 26 15:18:29 dist-test-slave-j6q7 krb5kdc[22430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.195@KRBTEST.COM
I20260926 15:18:29.948844 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:29.931157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34524 (local address 127.6.209.195:37809)
0926 15:18:29.931704 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:29.931721 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:29.931736 (+ 15us) server_negotiation.cc:407] Connection header received
0926 15:18:29.931788 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:29.931793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:29.931875 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:29.931984 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:29.934004 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.935139 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.936873 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:29.937311 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:29.937397 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:29.942621 (+ 5224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:29.942635 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:29.942653 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:29.942697 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:29.946347 (+ 3650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:29.946910 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:29.946916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:29.946918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:29.947052 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:29.947366 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:29.947371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:29.947374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:29.947687 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:29.947978 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:29.948275 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 15:18:29.948482 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431}
I20260926 15:18:29.949476 22926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:18:30.057957 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 22803
I20260926 15:18:30.075285 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 22443
I20260926 15:18:30.076310 22505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:18:30.178093 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 22443
I20260926 15:18:50.198504 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44575
--webserver_interface=127.6.209.254
--webserver_port=42513
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:44575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:18:50.490283 22980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:18:50.490559 22980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:18:50.490705 22980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:18:50.497022 22980 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 15:18:50.497133 22980 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 15:18:50.497221 22980 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 15:18:50.497354 22980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:18:50.497433 22980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:18:50.497524 22980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:18:50.497599 22980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:18:50.497673 22980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:18:50.505944 22980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:44575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=42513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:18:50.507046 22980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:18:50.508242 22980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:18:50.529708 22986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:50.529626 22987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:50.529683 22989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:18:50.531157 22980 server_base.cc:1034] running on GCE node
I20260926 15:18:50.531898 22980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:18:50.533332 22980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:18:50.534547 22980 hybrid_clock.cc:648] HybridClock initialized: now 1790435930534468 us; error 50 us; skew 500 ppm
I20260926 15:18:50.534860 22980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:18:50 dist-test-slave-j6q7 krb5kdc[22430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435930, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:18:50.542207 22980 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:18:50.543630 22980 webserver.cc:460] Webserver started at http://127.6.209.254:42513/ using document root <none> and password file <none>
I20260926 15:18:50.544292 22980 fs_manager.cc:359] Metadata directory not provided
I20260926 15:18:50.544399 22980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:18:50.549958 22980 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260926 15:18:50.552687 22996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:50.553961 22980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:18:50.554136 22980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c"
format_stamp: "Formatted at 2026-09-26 15:18:27 on dist-test-slave-j6q7"
server_key: "122b0d5561cf4baa693fcf0f9667b395"
server_key_iv: "32846de6a27f06857ea5e47a5488288a"
server_key_version: "encryptionkey@0"
I20260926 15:18:50.555119 22980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:18:50.579124 22980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:18:50.582545 22980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:18:50.582778 22980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:18:50.600481 22980 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:44575
I20260926 15:18:50.600553 23047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:44575 every 8 connection(s)
I20260926 15:18:50.601810 22980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:18:50.607792 23048 sys_catalog.cc:263] Verifying existing consensus state
I20260926 15:18:50.608312 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 22980
I20260926 15:18:50.609040 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:46439
--local_ip_for_outbound_sockets=127.6.209.193
--tserver_master_addrs=127.6.209.254:44575
--webserver_port=44863
--webserver_interface=127.6.209.193
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:18:50.610656 23048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c: Bootstrap starting.
I20260926 15:18:50.628511 23048 log.cc:824] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c: Log is configured to *not* fsync() on all Append() calls
I20260926 15:18:50.639776 23048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:18:50.640451 23048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c: Bootstrap complete.
I20260926 15:18:50.651259 23048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44575 } }
I20260926 15:18:50.651993 23048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8b515c2a034cbdb9c1d05bdd38c55c, State: Initialized, Role: FOLLOWER
I20260926 15:18:50.652609 23048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44575 } }
I20260926 15:18:50.652849 23048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:18:50.652951 23048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:18:50.653115 23048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 1 FOLLOWER]: Advancing to term 2
I20260926 15:18:50.656559 23048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44575 } }
I20260926 15:18:50.657069 23048 leader_election.cc:302] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f8b515c2a034cbdb9c1d05bdd38c55c; no voters:
I20260926 15:18:50.657680 23048 leader_election.cc:288] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 15:18:50.657787 23052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 2 FOLLOWER]: Leader election won for term 2
I20260926 15:18:50.659821 23048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:18:50.664888 23052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 2 LEADER]: Becoming Leader. State: Replica: 5f8b515c2a034cbdb9c1d05bdd38c55c, State: Running, Role: LEADER
I20260926 15:18:50.665464 23052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44575 } }
I20260926 15:18:50.668011 23053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44575 } } }
I20260926 15:18:50.668375 23053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [sys.catalog]: This master's current role is: LEADER
I20260926 15:18:50.668898 23054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f8b515c2a034cbdb9c1d05bdd38c55c. Latest consensus state: current_term: 2 leader_uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44575 } } }
I20260926 15:18:50.669190 23054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [sys.catalog]: This master's current role is: LEADER
I20260926 15:18:50.671008 23067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:18:50.686908 23067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:18:50.688090 23067 catalog_manager.cc:1259] Loaded cluster ID: 507c68b71ccf4dc2bd7ec78c4a97dab0
I20260926 15:18:50.688169 23067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:18:50.691471 23067 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:18:50.692701 23067 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c: Loaded TSK: 0
I20260926 15:18:50.718155 23067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c: Generated new TSK 1
I20260926 15:18:50.718544 23067 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 15:18:50.924500 23050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:18:50.924768 23050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:18:50.924949 23050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:18:50.931329 23050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:18:50.931456 23050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:18:50.931607 23050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:18:50.940759 23050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:46439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=44863
--tserver_master_addrs=127.6.209.254:44575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:18:50.941974 23050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:18:50.943150 23050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:18:50.956974 23076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:50.956946 23077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:18:50.957811 23050 server_base.cc:1034] running on GCE node
W20260926 15:18:50.958278 23079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:18:50.959007 23050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:18:50.959951 23050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:18:50.961199 23050 hybrid_clock.cc:648] HybridClock initialized: now 1790435930961125 us; error 73 us; skew 500 ppm
I20260926 15:18:50.961514 23050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:18:50 dist-test-slave-j6q7 krb5kdc[22430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435930, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:18:50.964421 23050 init.cc:377] Logged in from keytab as kudu/127.6.209.193@KRBTEST.COM (short username kudu)
I20260926 15:18:50.965873 23050 webserver.cc:460] Webserver started at http://127.6.209.193:44863/ using document root <none> and password file <none>
I20260926 15:18:50.966533 23050 fs_manager.cc:359] Metadata directory not provided
I20260926 15:18:50.966665 23050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:18:50.971920 23050 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 15:18:50.974534 23086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:50.975906 23050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 15:18:50.976060 23050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "bab30ea8c2ee4e2fa84ed656a989b777"
format_stamp: "Formatted at 2026-09-26 15:18:28 on dist-test-slave-j6q7"
server_key: "23a13246a18d746cf5642bf2b2c8c0c1"
server_key_iv: "1b5fde8d764dc8eebf53b26fd80d3490"
server_key_version: "encryptionkey@0"
I20260926 15:18:50.977051 23050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:18:50.991941 23050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:18:50.995913 23050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:18:50.996212 23050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:18:50.997219 23050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:18:50.998529 23050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:18:50.998648 23050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:50.998781 23050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:18:50.998880 23050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:51.033051 23050 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:46439
I20260926 15:18:51.033113 23198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:46439 every 8 connection(s)
I20260926 15:18:51.034430 23050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:18:51.039079 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 23050
I20260926 15:18:51.039847 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:34287
--local_ip_for_outbound_sockets=127.6.209.194
--tserver_master_addrs=127.6.209.254:44575
--webserver_port=42025
--webserver_interface=127.6.209.194
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:18:51 dist-test-slave-j6q7 krb5kdc[22430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435930, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:18:51.062114 23202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:51.037626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.193:40239)
0926 15:18:51.041359 (+ 3733us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:51.041403 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:51.042925 (+ 1522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:51.044125 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:51.044135 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:51.044710 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:18:51.045225 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:51.045239 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.046780 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.046790 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:51.048016 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:51.048030 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.049453 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.049460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:51.049729 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:51.050444 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:18:51.050472 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:18:51.053156 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:18:51.057609 (+ 4453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:18:51.057617 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:18:51.057628 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:18:51.058142 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:18:51.058704 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:18:51.058712 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:18:51.058718 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:18:51.058946 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:18:51.059735 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:18:51.059743 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:18:51.060087 (+ 344us) client_negotiation.cc:769] Sending connection context
0926 15:18:51.060385 (+ 298us) client_negotiation.cc:240] Negotiation successful
0926 15:18:51.060907 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3476,"thread_start_us":3284,"threads_started":1}
I20260926 15:18:51.062139 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:51.037394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:40239 (local address 127.6.209.254:44575)
0926 15:18:51.038089 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:51.038104 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:51.043182 (+ 5078us) server_negotiation.cc:407] Connection header received
0926 15:18:51.043424 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:51.043433 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:51.043723 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:51.044011 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:51.045414 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.046621 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.048185 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.049240 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.049411 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:51.053453 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:51.053464 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:51.053480 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:51.053523 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:51.057467 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:51.058357 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:51.058365 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:51.058372 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:51.058581 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:51.059067 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:51.059070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:51.059072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:51.059597 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:51.059792 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:51.060612 (+ 820us) server_negotiation.cc:299] Negotiation successful
0926 15:18:51.061007 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":313,"threads_started":1}
I20260926 15:18:51.064788 23199 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44575
I20260926 15:18:51.065157 23199 heartbeater.cc:464] Registering TS with master...
I20260926 15:18:51.065834 23199 heartbeater.cc:511] Master 127.6.209.254:44575 requested a full tablet report, sending...
I20260926 15:18:51.068003 23013 ts_manager.cc:194] Registered new tserver with Master: bab30ea8c2ee4e2fa84ed656a989b777 (127.6.209.193:46439)
I20260926 15:18:51.069813 23013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.193@KRBTEST.COM'} at 127.6.209.193:40239
W20260926 15:18:51.336057 23203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:18:51.336349 23203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:18:51.336443 23203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:18:51.343922 23203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:18:51.344033 23203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:18:51.344146 23203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:18:51.353340 23203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:34287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=42025
--tserver_master_addrs=127.6.209.254:44575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:18:51.354434 23203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:18:51.355577 23203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:18:51.368561 23208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:51.368498 23209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:51.368530 23211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:18:51.369113 23203 server_base.cc:1034] running on GCE node
I20260926 15:18:51.369710 23203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:18:51.370599 23203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:18:51.371815 23203 hybrid_clock.cc:648] HybridClock initialized: now 1790435931371779 us; error 47 us; skew 500 ppm
I20260926 15:18:51.372118 23203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:18:51 dist-test-slave-j6q7 krb5kdc[22430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435931, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:18:51.374970 23203 init.cc:377] Logged in from keytab as kudu/127.6.209.194@KRBTEST.COM (short username kudu)
I20260926 15:18:51.376346 23203 webserver.cc:460] Webserver started at http://127.6.209.194:42025/ using document root <none> and password file <none>
I20260926 15:18:51.377054 23203 fs_manager.cc:359] Metadata directory not provided
I20260926 15:18:51.377206 23203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:18:51.382227 23203 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 15:18:51.384634 23218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:51.385842 23203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:18:51.386013 23203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "b0f73802c54b46b09e4d723173fc3806"
format_stamp: "Formatted at 2026-09-26 15:18:28 on dist-test-slave-j6q7"
server_key: "5643f667f15ca64af786e09314fd95bd"
server_key_iv: "47359fd8b7326e0f8ce53018dc4cc8f7"
server_key_version: "encryptionkey@0"
I20260926 15:18:51.386914 23203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:18:51.415787 23203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:18:51.419363 23203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:18:51.419613 23203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:18:51.420485 23203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:18:51.421739 23203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:18:51.421877 23203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:51.421993 23203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:18:51.422076 23203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:51.453401 23203 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:34287
I20260926 15:18:51.453533 23330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:34287 every 8 connection(s)
I20260926 15:18:51.454681 23203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:18:51.463917 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 23203
I20260926 15:18:51.464623 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:37809
--local_ip_for_outbound_sockets=127.6.209.195
--tserver_master_addrs=127.6.209.254:44575
--webserver_port=33707
--webserver_interface=127.6.209.195
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:18:51 dist-test-slave-j6q7 krb5kdc[22430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435931, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:18:51.474000 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:51.457051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:42257 (local address 127.6.209.254:44575)
0926 15:18:51.457220 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:51.457224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:51.458362 (+ 1138us) server_negotiation.cc:407] Connection header received
0926 15:18:51.460208 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:51.460213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:51.460298 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:51.460415 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:51.461814 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.462526 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.464429 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.464953 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.465047 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:51.468780 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:51.468784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:51.468787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:51.468861 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:51.471181 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:51.471844 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:51.471850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:51.471855 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:51.471996 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:51.472389 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:51.472395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:51.472398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:51.472645 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:51.472796 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:51.473427 (+ 631us) server_negotiation.cc:299] Negotiation successful
0926 15:18:51.473648 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 15:18:51.474833 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:51.457543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.194:42257)
0926 15:18:51.458184 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:51.458245 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:51.459934 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:51.460632 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:51.460639 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:51.461168 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:18:51.461663 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:51.461678 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.462708 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.462713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:51.464280 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:51.464298 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.465156 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.465165 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:51.465469 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:51.466138 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:18:51.466168 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:18:51.468584 (+ 2416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:18:51.471317 (+ 2733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:18:51.471325 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:18:51.471338 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:18:51.471722 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:18:51.472102 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:18:51.472107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:18:51.472111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:18:51.472279 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:18:51.472772 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:18:51.472779 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:18:51.473078 (+ 299us) client_negotiation.cc:769] Sending connection context
0926 15:18:51.473341 (+ 263us) client_negotiation.cc:240] Negotiation successful
0926 15:18:51.473778 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":240,"threads_started":1}
I20260926 15:18:51.476240 23331 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44575
I20260926 15:18:51.476521 23331 heartbeater.cc:464] Registering TS with master...
I20260926 15:18:51.477191 23331 heartbeater.cc:511] Master 127.6.209.254:44575 requested a full tablet report, sending...
I20260926 15:18:51.478456 23013 ts_manager.cc:194] Registered new tserver with Master: b0f73802c54b46b09e4d723173fc3806 (127.6.209.194:34287)
I20260926 15:18:51.479177 23013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.194@KRBTEST.COM'} at 127.6.209.194:42257
W20260926 15:18:51.780386 23334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:18:51.780663 23334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:18:51.780846 23334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:18:51.787546 23334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:18:51.787689 23334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:18:51.787909 23334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:18:51.796859 23334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:37809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=33707
--tserver_master_addrs=127.6.209.254:44575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:18:51.797962 23334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:18:51.799127 23334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:18:51.811228 23339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:51.811256 23342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:18:51.811275 23340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:18:51.812080 23334 server_base.cc:1034] running on GCE node
I20260926 15:18:51.812520 23334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:18:51.813342 23334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:18:51.814522 23334 hybrid_clock.cc:648] HybridClock initialized: now 1790435931814490 us; error 27 us; skew 500 ppm
I20260926 15:18:51.814788 23334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:18:51 dist-test-slave-j6q7 krb5kdc[22430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435931, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:18:51.817610 23334 init.cc:377] Logged in from keytab as kudu/127.6.209.195@KRBTEST.COM (short username kudu)
I20260926 15:18:51.818768 23334 webserver.cc:460] Webserver started at http://127.6.209.195:33707/ using document root <none> and password file <none>
I20260926 15:18:51.819264 23334 fs_manager.cc:359] Metadata directory not provided
I20260926 15:18:51.819358 23334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:18:51.823812 23334 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:18:51.826081 23349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:51.827260 23334 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 15:18:51.827443 23334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "85783935ae1944c7afbb471f87c8abef"
format_stamp: "Formatted at 2026-09-26 15:18:29 on dist-test-slave-j6q7"
server_key: "f331207c1de312149c3bf1cfba35dc93"
server_key_iv: "b470db8e3f3ece21b33824a04d96bfc1"
server_key_version: "encryptionkey@0"
I20260926 15:18:51.828293 23334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:18:51.843834 23334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:18:51.847151 23334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:18:51.847394 23334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:18:51.848182 23334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:18:51.849299 23334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:18:51.849413 23334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:51.849539 23334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:18:51.849617 23334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:18:51.879354 23334 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:37809
I20260926 15:18:51.879429 23461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:37809 every 8 connection(s)
I20260926 15:18:51.880798 23334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:18:51.887568 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 23334
Sep 26 15:18:51 dist-test-slave-j6q7 krb5kdc[22430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435931, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:18:51.899462 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:51.883097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:41467 (local address 127.6.209.254:44575)
0926 15:18:51.883340 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:51.883345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:51.884402 (+ 1057us) server_negotiation.cc:407] Connection header received
0926 15:18:51.885939 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:51.885943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:51.886001 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:51.886081 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:51.887435 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.888339 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.890407 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.890842 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.890966 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:51.894410 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:51.894414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:51.894419 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:51.894449 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:51.896766 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:51.897452 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:51.897457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:51.897460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:51.897599 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:51.897992 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:51.897996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:51.897998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:51.898207 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:51.898372 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:51.898881 (+ 509us) server_negotiation.cc:299] Negotiation successful
0926 15:18:51.899139 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260926 15:18:51.900125 23464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:51.883556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.195:41467)
0926 15:18:51.884258 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:51.884294 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:51.885703 (+ 1409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:51.886295 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:51.886302 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:51.886747 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:18:51.887245 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:51.887260 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.888504 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.888509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:51.890278 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:51.890293 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.891066 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.891074 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:51.891248 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:51.891779 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:18:51.891797 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:18:51.894197 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:18:51.896912 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:18:51.896917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:18:51.896946 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:18:51.897318 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:18:51.897699 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:18:51.897705 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:18:51.897708 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:18:51.897872 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:18:51.898335 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:18:51.898341 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:18:51.898585 (+ 244us) client_negotiation.cc:769] Sending connection context
0926 15:18:51.898792 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 15:18:51.899177 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":239,"threads_started":1}
I20260926 15:18:51.901592 23462 heartbeater.cc:347] Connected to a master server at 127.6.209.254:44575
I20260926 15:18:51.901815 23466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:51.888412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55628 (local address 127.6.209.254:44575)
0926 15:18:51.889066 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:51.889069 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:51.889081 (+ 12us) server_negotiation.cc:407] Connection header received
0926 15:18:51.889113 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:51.889115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:51.889156 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:51.889229 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:51.891078 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.891954 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.893847 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.894181 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.894259 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:51.896643 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:51.896649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:51.896653 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:51.896702 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:51.899233 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:51.899881 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:51.899885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:51.899888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:51.900016 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:51.900416 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:51.900420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:51.900422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:51.900754 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:51.901011 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:51.901303 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 15:18:51.901457 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":476,"threads_started":1}
I20260926 15:18:51.901891 23462 heartbeater.cc:464] Registering TS with master...
I20260926 15:18:51.902540 23462 heartbeater.cc:511] Master 127.6.209.254:44575 requested a full tablet report, sending...
I20260926 15:18:51.903893 23012 ts_manager.cc:194] Registered new tserver with Master: 85783935ae1944c7afbb471f87c8abef (127.6.209.195:37809)
I20260926 15:18:51.904619 23012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.195@KRBTEST.COM'} at 127.6.209.195:41467
I20260926 15:18:51.915657 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:18:51.921600 23466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:51.916442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55638 (local address 127.6.209.254:44575)
0926 15:18:51.916589 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:51.916596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:51.917037 (+ 441us) server_negotiation.cc:407] Connection header received
0926 15:18:51.917242 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:51.917246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:51.917373 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:51.917517 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:18:51.917881 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.918479 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.919782 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.920199 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.920349 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:51.920854 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:18:51.921025 (+ 171us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 15:18:51.921369 (+ 344us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55638: token expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 15:18:51.921711 23466 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55638: token expired
W20260926 15:18:51.922070 22935 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.209.254:44575)
I20260926 15:18:51.932081 23466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:51.922445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55644 (local address 127.6.209.254:44575)
0926 15:18:51.922584 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:51.922587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:51.922600 (+ 13us) server_negotiation.cc:407] Connection header received
0926 15:18:51.922704 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:51.922706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:51.922746 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:51.922808 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:51.923791 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.924337 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.925453 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.925908 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.925988 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:51.928050 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:51.928053 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:51.928056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:51.928077 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:51.930163 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:51.930620 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:51.930623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:51.930625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:51.930721 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:51.931011 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:51.931014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:51.931015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:51.931278 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:51.931430 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:51.931701 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 15:18:51.931824 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 15:18:51.934486 23012 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:18:51.941130 23012 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:18:51.963483 23474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:51.951336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:34287 (local address 127.0.0.1:47274)
0926 15:18:51.955242 (+ 3906us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:51.955263 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:51.955395 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:51.956518 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:51.956524 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:51.956558 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:18:51.956909 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:51.956922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.958567 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.958574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:51.960585 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:51.960600 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.962658 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.962665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:51.962845 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:51.962914 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 15:18:51.963030 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 15:18:51.963182 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3779,"thread_start_us":3654,"threads_started":1}
I20260926 15:18:51.963783 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:51.952160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.6.209.194:34287)
0926 15:18:51.955851 (+ 3691us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:51.955860 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:51.955893 (+ 33us) server_negotiation.cc:407] Connection header received
0926 15:18:51.955988 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:51.955996 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:51.956212 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:51.956418 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:18:51.957104 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.958452 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.960744 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.962212 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.962315 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:51.962532 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:51.963101 (+ 569us) server_negotiation.cc:299] Negotiation successful
0926 15:18:51.963250 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3545,"thread_start_us":3427,"threads_started":1}
I20260926 15:18:51.964623 23472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:51.950705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:37809 (local address 127.0.0.1:45182)
0926 15:18:51.952056 (+ 1351us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:51.952089 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:51.952217 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:51.957150 (+ 4933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:51.957154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:51.957195 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:18:51.958101 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:51.958111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.959879 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.959884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:51.962108 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:51.962122 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.963881 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.963888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:51.964055 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:51.964086 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 15:18:51.964151 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 15:18:51.964265 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":399,"spinlock_wait_cycles":7808,"thread_start_us":325,"threads_started":1}
I20260926 15:18:51.965010 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:51.951761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:46439 (local address 127.0.0.1:44562)
0926 15:18:51.956201 (+ 4440us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:51.956236 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:51.956412 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:51.957150 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:51.957154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:51.957196 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:18:51.957883 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:51.957893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.959665 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.959670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:51.961998 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:51.962008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.964174 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.964178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:51.964280 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:51.964299 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 15:18:51.964401 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 15:18:51.964511 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4245,"mutex_wait_us":91,"spinlock_wait_cycles":7808,"thread_start_us":4138,"threads_started":1}
I20260926 15:18:51.965039 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:51.952247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44562 (local address 127.6.209.193:46439)
0926 15:18:51.952884 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:51.952894 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:51.956318 (+ 3424us) server_negotiation.cc:407] Connection header received
0926 15:18:51.956593 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:51.956600 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:51.956792 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:51.956970 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:18:51.958085 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.959536 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.962257 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.963739 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.963868 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:51.964041 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:51.964443 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 15:18:51.964577 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":328,"threads_started":1}
I20260926 15:18:51.965481 23473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:51.951018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45182 (local address 127.6.209.195:37809)
0926 15:18:51.956565 (+ 5547us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:51.956572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:51.956604 (+ 32us) server_negotiation.cc:407] Connection header received
0926 15:18:51.956689 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:51.956698 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:51.956953 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:51.957265 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:18:51.958308 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.959754 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.962304 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:51.963648 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:51.964841 (+ 1193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:51.965035 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:51.965092 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 15:18:51.965212 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5403,"thread_start_us":5243,"threads_started":1}
I20260926 15:18:51.967188 23266 tablet_service.cc:1467] Processing CreateTablet for tablet 5aaf2666eb70435fbad886cf6402fcc1 (DEFAULT_TABLE table=test-table [id=5fe9453346f14922bc746663f94813b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:18:51.968294 23266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aaf2666eb70435fbad886cf6402fcc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:18:51.968312 23397 tablet_service.cc:1467] Processing CreateTablet for tablet 5aaf2666eb70435fbad886cf6402fcc1 (DEFAULT_TABLE table=test-table [id=5fe9453346f14922bc746663f94813b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:18:51.968312 23134 tablet_service.cc:1467] Processing CreateTablet for tablet 5aaf2666eb70435fbad886cf6402fcc1 (DEFAULT_TABLE table=test-table [id=5fe9453346f14922bc746663f94813b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:18:51.969450 23397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aaf2666eb70435fbad886cf6402fcc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:18:51.969462 23134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aaf2666eb70435fbad886cf6402fcc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:18:51.978322 23478 tablet_bootstrap.cc:492] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806: Bootstrap starting.
I20260926 15:18:51.981312 23478 tablet_bootstrap.cc:654] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806: Neither blocks nor log segments found. Creating new log.
I20260926 15:18:51.982225 23478 log.cc:824] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806: Log is configured to *not* fsync() on all Append() calls
I20260926 15:18:51.982709 23479 tablet_bootstrap.cc:492] T 5aaf2666eb70435fbad886cf6402fcc1 P 85783935ae1944c7afbb471f87c8abef: Bootstrap starting.
I20260926 15:18:51.983752 23481 tablet_bootstrap.cc:492] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777: Bootstrap starting.
I20260926 15:18:51.986382 23478 tablet_bootstrap.cc:492] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806: No bootstrap required, opened a new log
I20260926 15:18:51.986500 23479 tablet_bootstrap.cc:654] T 5aaf2666eb70435fbad886cf6402fcc1 P 85783935ae1944c7afbb471f87c8abef: Neither blocks nor log segments found. Creating new log.
I20260926 15:18:51.986639 23478 ts_tablet_manager.cc:1397] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 15:18:51.986763 23481 tablet_bootstrap.cc:654] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777: Neither blocks nor log segments found. Creating new log.
I20260926 15:18:51.987632 23481 log.cc:824] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777: Log is configured to *not* fsync() on all Append() calls
I20260926 15:18:51.987658 23479 log.cc:824] T 5aaf2666eb70435fbad886cf6402fcc1 P 85783935ae1944c7afbb471f87c8abef: Log is configured to *not* fsync() on all Append() calls
I20260926 15:18:51.989838 23481 tablet_bootstrap.cc:492] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777: No bootstrap required, opened a new log
I20260926 15:18:51.990039 23481 ts_tablet_manager.cc:1397] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 15:18:51.997141 23481 raft_consensus.cc:348] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } } peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } }
I20260926 15:18:51.997586 23481 raft_consensus.cc:374] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:18:51.997694 23481 raft_consensus.cc:729] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bab30ea8c2ee4e2fa84ed656a989b777, State: Initialized, Role: FOLLOWER
I20260926 15:18:51.998270 23481 consensus_queue.cc:260] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } } peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } }
I20260926 15:18:51.998790 23478 raft_consensus.cc:348] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } } peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } }
I20260926 15:18:51.999171 23478 raft_consensus.cc:374] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:18:51.999248 23478 raft_consensus.cc:729] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f73802c54b46b09e4d723173fc3806, State: Initialized, Role: FOLLOWER
I20260926 15:18:51.999266 23479 tablet_bootstrap.cc:492] T 5aaf2666eb70435fbad886cf6402fcc1 P 85783935ae1944c7afbb471f87c8abef: No bootstrap required, opened a new log
I20260926 15:18:51.999492 23479 ts_tablet_manager.cc:1397] T 5aaf2666eb70435fbad886cf6402fcc1 P 85783935ae1944c7afbb471f87c8abef: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.014s
I20260926 15:18:51.999605 23199 heartbeater.cc:503] Master 127.6.209.254:44575 was elected leader, sending a full tablet report...
I20260926 15:18:51.999995 23481 ts_tablet_manager.cc:1428] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260926 15:18:52.001022 23478 consensus_queue.cc:260] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } } peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } }
I20260926 15:18:52.001957 23331 heartbeater.cc:503] Master 127.6.209.254:44575 was elected leader, sending a full tablet report...
I20260926 15:18:52.002539 23478 ts_tablet_manager.cc:1428] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260926 15:18:52.008023 23479 raft_consensus.cc:348] T 5aaf2666eb70435fbad886cf6402fcc1 P 85783935ae1944c7afbb471f87c8abef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } } peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } }
I20260926 15:18:52.008318 23479 raft_consensus.cc:374] T 5aaf2666eb70435fbad886cf6402fcc1 P 85783935ae1944c7afbb471f87c8abef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:18:52.008442 23479 raft_consensus.cc:729] T 5aaf2666eb70435fbad886cf6402fcc1 P 85783935ae1944c7afbb471f87c8abef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85783935ae1944c7afbb471f87c8abef, State: Initialized, Role: FOLLOWER
I20260926 15:18:52.008963 23479 consensus_queue.cc:260] T 5aaf2666eb70435fbad886cf6402fcc1 P 85783935ae1944c7afbb471f87c8abef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } } peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } }
I20260926 15:18:52.010056 23462 heartbeater.cc:503] Master 127.6.209.254:44575 was elected leader, sending a full tablet report...
I20260926 15:18:52.010466 23479 ts_tablet_manager.cc:1428] T 5aaf2666eb70435fbad886cf6402fcc1 P 85783935ae1944c7afbb471f87c8abef: Time spent starting tablet: real 0.011s user 0.003s sys 0.008s
W20260926 15:18:52.036533 23200 tablet.cc:2369] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:18:52.132402 23463 tablet.cc:2369] T 5aaf2666eb70435fbad886cf6402fcc1 P 85783935ae1944c7afbb471f87c8abef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:18:52.148530 23485 raft_consensus.cc:482] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:18:52.148800 23485 raft_consensus.cc:504] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } } peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } }
I20260926 15:18:52.150399 23485 leader_election.cc:288] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85783935ae1944c7afbb471f87c8abef (127.6.209.195:37809), bab30ea8c2ee4e2fa84ed656a989b777 (127.6.209.193:46439)
I20260926 15:18:52.156179 23473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:52.150690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:46947 (local address 127.6.209.195:37809)
0926 15:18:52.150859 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:52.150863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:52.151397 (+ 534us) server_negotiation.cc:407] Connection header received
0926 15:18:52.151590 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:52.151598 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:52.151682 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:52.151803 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:18:52.152416 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:52.153134 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:52.154630 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:52.155349 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:52.155515 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:52.155582 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:52.155812 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 15:18:52.155930 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 15:18:52.156315 23487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:52.150751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:37809 (local address 127.6.209.194:46947)
0926 15:18:52.151290 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:52.151304 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:52.151468 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:52.151852 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:52.151857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:52.151895 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:18:52.152283 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:52.152293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:52.153256 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:52.153260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:52.154504 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:52.154514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:52.155536 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:52.155542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:52.155672 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:52.155693 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 15:18:52.155788 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 15:18:52.155905 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":19,"thread_start_us":284,"threads_started":1}
I20260926 15:18:52.156597 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:52.151605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:36739 (local address 127.6.209.193:46439)
0926 15:18:52.151818 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:52.151832 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:52.152089 (+ 257us) server_negotiation.cc:407] Connection header received
0926 15:18:52.152260 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:52.152271 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:52.152324 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:52.152411 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:18:52.152889 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:52.153486 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:52.154962 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:52.155669 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:52.155845 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:52.155908 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:52.156183 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 15:18:52.156346 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 15:18:52.156975 23488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:52.151376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:46439 (local address 127.6.209.194:36739)
0926 15:18:52.151998 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:52.152010 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:18:52.152134 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:18:52.152427 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:18:52.152431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:18:52.152452 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:18:52.152729 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:52.152737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:52.153609 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:52.153613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:52.154808 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:18:52.154818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:18:52.155859 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:18:52.155864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:18:52.156023 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:52.156043 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 15:18:52.156155 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 15:18:52.156584 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":22,"thread_start_us":740,"threads_started":1}
I20260926 15:18:52.157325 23417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aaf2666eb70435fbad886cf6402fcc1" candidate_uuid: "b0f73802c54b46b09e4d723173fc3806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85783935ae1944c7afbb471f87c8abef" is_pre_election: true
I20260926 15:18:52.157748 23417 raft_consensus.cc:2454] T 5aaf2666eb70435fbad886cf6402fcc1 P 85783935ae1944c7afbb471f87c8abef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0f73802c54b46b09e4d723173fc3806 in term 0.
I20260926 15:18:52.157871 23154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aaf2666eb70435fbad886cf6402fcc1" candidate_uuid: "b0f73802c54b46b09e4d723173fc3806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" is_pre_election: true
I20260926 15:18:52.158337 23222 leader_election.cc:302] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85783935ae1944c7afbb471f87c8abef, b0f73802c54b46b09e4d723173fc3806; no voters:
I20260926 15:18:52.158391 23154 raft_consensus.cc:2454] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0f73802c54b46b09e4d723173fc3806 in term 0.
I20260926 15:18:52.158674 23485 raft_consensus.cc:2781] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:18:52.158797 23485 raft_consensus.cc:482] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:18:52.158900 23485 raft_consensus.cc:3037] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:18:52.161216 23485 raft_consensus.cc:504] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } } peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } }
I20260926 15:18:52.161736 23485 leader_election.cc:288] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806 [CANDIDATE]: Term 1 election: Requested vote from peers 85783935ae1944c7afbb471f87c8abef (127.6.209.195:37809), bab30ea8c2ee4e2fa84ed656a989b777 (127.6.209.193:46439)
I20260926 15:18:52.162048 23417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aaf2666eb70435fbad886cf6402fcc1" candidate_uuid: "b0f73802c54b46b09e4d723173fc3806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85783935ae1944c7afbb471f87c8abef"
I20260926 15:18:52.162118 23154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aaf2666eb70435fbad886cf6402fcc1" candidate_uuid: "b0f73802c54b46b09e4d723173fc3806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bab30ea8c2ee4e2fa84ed656a989b777"
I20260926 15:18:52.162211 23417 raft_consensus.cc:3037] T 5aaf2666eb70435fbad886cf6402fcc1 P 85783935ae1944c7afbb471f87c8abef [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:18:52.162308 23154 raft_consensus.cc:3037] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:18:52.164346 23154 raft_consensus.cc:2454] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0f73802c54b46b09e4d723173fc3806 in term 1.
I20260926 15:18:52.164363 23417 raft_consensus.cc:2454] T 5aaf2666eb70435fbad886cf6402fcc1 P 85783935ae1944c7afbb471f87c8abef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0f73802c54b46b09e4d723173fc3806 in term 1.
I20260926 15:18:52.164681 23222 leader_election.cc:302] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85783935ae1944c7afbb471f87c8abef, b0f73802c54b46b09e4d723173fc3806; no voters:
I20260926 15:18:52.164923 23485 raft_consensus.cc:2781] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:18:52.165275 23485 raft_consensus.cc:686] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806 [term 1 LEADER]: Becoming Leader. State: Replica: b0f73802c54b46b09e4d723173fc3806, State: Running, Role: LEADER
I20260926 15:18:52.165648 23485 consensus_queue.cc:237] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } } peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } }
I20260926 15:18:52.169620 23013 catalog_manager.cc:5495] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0f73802c54b46b09e4d723173fc3806 (127.6.209.194). New cstate: current_term: 1 leader_uuid: "b0f73802c54b46b09e4d723173fc3806" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } health_report { overall_health: HEALTHY } } }
I20260926 15:18:52.198217 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:52.193534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47276 (local address 127.6.209.194:34287)
0926 15:18:52.193705 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:52.193709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:52.193821 (+ 112us) server_negotiation.cc:407] Connection header received
0926 15:18:52.193960 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:52.193963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:52.194032 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:52.194140 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:18:52.194426 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:52.195055 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:52.196432 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:52.196880 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:52.196968 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:52.197425 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:18:52.197848 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:18:52.197940 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:52.198004 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 15:18:52.198057 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 15:18:52.205382 23154 raft_consensus.cc:1261] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Refusing update from remote peer b0f73802c54b46b09e4d723173fc3806: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:18:52.205396 23417 raft_consensus.cc:1261] T 5aaf2666eb70435fbad886cf6402fcc1 P 85783935ae1944c7afbb471f87c8abef [term 1 FOLLOWER]: Refusing update from remote peer b0f73802c54b46b09e4d723173fc3806: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:18:52.206178 23489 consensus_queue.cc:1035] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 15:18:52.206492 23332 tablet.cc:2369] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:18:52.206553 23485 consensus_queue.cc:1035] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:18:52.228163 23012 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:18:52.228387 23012 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:18:52.231837 23012 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c: Sending DeleteTablet for 3 replicas of tablet 5aaf2666eb70435fbad886cf6402fcc1
I20260926 15:18:52.233023 23134 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aaf2666eb70435fbad886cf6402fcc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:18:52 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:44562
I20260926 15:18:52.233026 23397 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aaf2666eb70435fbad886cf6402fcc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:18:52 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:45182
I20260926 15:18:52.233228 23266 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aaf2666eb70435fbad886cf6402fcc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:18:52 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:47274
I20260926 15:18:52.234202 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 23050
I20260926 15:18:52.243826 23503 tablet_replica.cc:330] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806: stopping tablet replica
I20260926 15:18:52.244333 23503 raft_consensus.cc:2229] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:18:52.244989 23503 raft_consensus.cc:2258] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:18:52.245213 23501 tablet_replica.cc:330] T 5aaf2666eb70435fbad886cf6402fcc1 P 85783935ae1944c7afbb471f87c8abef: stopping tablet replica
I20260926 15:18:52.245631 23501 raft_consensus.cc:2229] T 5aaf2666eb70435fbad886cf6402fcc1 P 85783935ae1944c7afbb471f87c8abef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:18:52.246158 23501 raft_consensus.cc:2258] T 5aaf2666eb70435fbad886cf6402fcc1 P 85783935ae1944c7afbb471f87c8abef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:18:52.248250 23501 ts_tablet_manager.cc:1905] T 5aaf2666eb70435fbad886cf6402fcc1 P 85783935ae1944c7afbb471f87c8abef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:18:52.250942 23502 tablet_replica.cc:330] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777: stopping tablet replica
I20260926 15:18:52.251263 23502 raft_consensus.cc:2229] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:18:52.251672 23502 raft_consensus.cc:2258] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:18:52.253661 23502 ts_tablet_manager.cc:1905] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:18:52.254287 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:52.235480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44570 (local address 127.6.209.193:46439)
0926 15:18:52.235652 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:52.235657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:52.235672 (+ 15us) server_negotiation.cc:407] Connection header received
0926 15:18:52.235720 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:52.235725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:52.235790 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:52.235899 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:52.237796 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:52.238867 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:52.240883 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:52.241539 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:52.241645 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:52.246122 (+ 4477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:52.246135 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:52.246148 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:52.246188 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:52.250363 (+ 4175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:52.251186 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:52.251193 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:52.251197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:52.251334 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:52.251891 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:52.251897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:52.251900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:52.252354 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:52.252639 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:52.253703 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0926 15:18:52.253963 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 15:18:52.254951 23501 ts_tablet_manager.cc:1918] T 5aaf2666eb70435fbad886cf6402fcc1 P 85783935ae1944c7afbb471f87c8abef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:18:52.255075 23501 log.cc:1196] T 5aaf2666eb70435fbad886cf6402fcc1 P 85783935ae1944c7afbb471f87c8abef: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/5aaf2666eb70435fbad886cf6402fcc1
I20260926 15:18:52.255076 23194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:18:52.255506 23501 ts_tablet_manager.cc:1939] T 5aaf2666eb70435fbad886cf6402fcc1 P 85783935ae1944c7afbb471f87c8abef: Deleting consensus metadata
I20260926 15:18:52.256927 23000 catalog_manager.cc:4841] TS 85783935ae1944c7afbb471f87c8abef (127.6.209.195:37809): tablet 5aaf2666eb70435fbad886cf6402fcc1 (table test-table [id=5fe9453346f14922bc746663f94813b0]) successfully deleted
I20260926 15:18:52.258082 23503 ts_tablet_manager.cc:1905] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:18:52.265192 23503 ts_tablet_manager.cc:1918] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:18:52.265339 23503 log.cc:1196] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/5aaf2666eb70435fbad886cf6402fcc1
I20260926 15:18:52.265829 23503 ts_tablet_manager.cc:1939] T 5aaf2666eb70435fbad886cf6402fcc1 P b0f73802c54b46b09e4d723173fc3806: Deleting consensus metadata
I20260926 15:18:52.267493 22997 catalog_manager.cc:4841] TS b0f73802c54b46b09e4d723173fc3806 (127.6.209.194:34287): tablet 5aaf2666eb70435fbad886cf6402fcc1 (table test-table [id=5fe9453346f14922bc746663f94813b0]) successfully deleted
I20260926 15:18:52.342694 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 23050
I20260926 15:18:52.359162 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 23203
W20260926 15:18:52.359313 23000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.209.193:46439 (error 108)
I20260926 15:18:52.362754 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:52.361399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:46439 (local address 127.0.0.1:44580)
0926 15:18:52.362460 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:52.362565 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":931}
W20260926 15:18:52.363013 23000 catalog_manager.cc:4584] TS bab30ea8c2ee4e2fa84ed656a989b777 (127.6.209.193:46439): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5aaf2666eb70435fbad886cf6402fcc1: Network error: Client connection negotiation failed: client connection to 127.6.209.193:46439: connect: Connection refused (error 111)
I20260926 15:18:52.373323 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:52.359991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47280 (local address 127.6.209.194:34287)
0926 15:18:52.360247 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:52.360255 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:52.360281 (+ 26us) server_negotiation.cc:407] Connection header received
0926 15:18:52.360477 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:52.360485 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:52.360593 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:52.360700 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:52.362336 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:52.363529 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:52.364990 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:52.365398 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:52.365535 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:52.368019 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:52.368030 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:52.368045 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:52.368079 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:52.370746 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:52.371418 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:52.371426 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:52.371429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:52.371605 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:52.371955 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:52.371960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:52.371962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:52.372272 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:52.372571 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:52.372793 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 15:18:52.372997 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260926 15:18:52.373937 23326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:18:52.416265 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:52.415917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:46439 (local address 127.0.0.1:44594)
0926 15:18:52.416075 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:52.416151 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 15:18:52.417290 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:52.416964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:46439 (local address 127.0.0.1:44602)
0926 15:18:52.417104 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:52.417183 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 15:18:52.472553 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 23203
I20260926 15:18:52.477619 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:52.477216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:46439 (local address 127.0.0.1:44612)
0926 15:18:52.477407 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:52.477504 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 15:18:52.478492 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:52.478196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:46439 (local address 127.0.0.1:44624)
0926 15:18:52.478339 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:52.478401 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 15:18:52.490713 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 23334
I20260926 15:18:52.506461 23473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:52.495077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45186 (local address 127.6.209.195:37809)
0926 15:18:52.495222 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 15:18:52.495226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:18:52.495242 (+ 16us) server_negotiation.cc:407] Connection header received
0926 15:18:52.495376 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:18:52.495379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:18:52.495440 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:18:52.495520 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:18:52.496800 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:52.497606 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:52.498620 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:18:52.498940 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:18:52.499017 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:18:52.501454 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:18:52.501465 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:18:52.501478 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:18:52.501522 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:18:52.504136 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:52.504752 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:52.504757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:52.504759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:52.504958 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:18:52.505282 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:18:52.505285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:18:52.505287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:18:52.505570 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:18:52.505747 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 15:18:52.505980 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 15:18:52.506120 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 15:18:52.507056 23457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:18:52.558859 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:52.558361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:46439 (local address 127.0.0.1:44628)
0926 15:18:52.558539 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:52.558650 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 15:18:52.560103 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:18:52.559632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:46439 (local address 127.0.0.1:44634)
0926 15:18:52.559838 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 15:18:52.559984 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260926 15:18:52.586840 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 23334
I20260926 15:18:52.603874 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 22980
I20260926 15:18:52.604926 23043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:18:52.711561 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 22980
I20260926 15:19:12.731590 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44575
--webserver_interface=127.6.209.254
--webserver_port=42513
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:44575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:19:13.027271 23509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:13.027508 23509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:13.027582 23509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:13.034014 23509 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 15:19:13.034090 23509 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 15:19:13.034170 23509 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 15:19:13.034268 23509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:19:13.034356 23509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:13.034466 23509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:13.034538 23509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:19:13.034608 23509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:19:13.043255 23509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:44575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:44575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=42513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:13.044282 23509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:13.045433 23509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:13.055842 23514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:13.055842 23517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:13.055846 23515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:13.056445 23509 server_base.cc:1034] running on GCE node
I20260926 15:19:13.057140 23509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:13.058215 23509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:13.059425 23509 hybrid_clock.cc:648] HybridClock initialized: now 1790435953059359 us; error 36 us; skew 500 ppm
I20260926 15:19:13.059773 23509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:19:13 dist-test-slave-j6q7 krb5kdc[22430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435953, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:19:13.066592 23509 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:19:13.067817 23509 webserver.cc:460] Webserver started at http://127.6.209.254:42513/ using document root <none> and password file <none>
I20260926 15:19:13.068393 23509 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:13.068521 23509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:13.073164 23509 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:19:13.075541 23524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:13.076588 23509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:19:13.076747 23509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c"
format_stamp: "Formatted at 2026-09-26 15:18:27 on dist-test-slave-j6q7"
server_key: "122b0d5561cf4baa693fcf0f9667b395"
server_key_iv: "32846de6a27f06857ea5e47a5488288a"
server_key_version: "encryptionkey@0"
I20260926 15:19:13.077615 23509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:13.097373 23509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:13.101300 23509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:13.101531 23509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:13.117730 23509 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:44575
I20260926 15:19:13.117794 23575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:44575 every 8 connection(s)
I20260926 15:19:13.119007 23509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:19:13.121827 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 23509
I20260926 15:19:13.122694 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:46439
--local_ip_for_outbound_sockets=127.6.209.193
--tserver_master_addrs=127.6.209.254:44575
--webserver_port=44863
--webserver_interface=127.6.209.193
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:19:13.126542 23576 sys_catalog.cc:263] Verifying existing consensus state
I20260926 15:19:13.131198 23576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c: Bootstrap starting.
I20260926 15:19:13.149214 23576 log.cc:824] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:13.163506 23576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:19:13.163929 23576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c: Bootstrap complete.
I20260926 15:19:13.171741 23576 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44575 } }
I20260926 15:19:13.172506 23576 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8b515c2a034cbdb9c1d05bdd38c55c, State: Initialized, Role: FOLLOWER
I20260926 15:19:13.173056 23576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44575 } }
I20260926 15:19:13.173250 23576 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:19:13.173348 23576 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:19:13.173504 23576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 2 FOLLOWER]: Advancing to term 3
I20260926 15:19:13.176705 23576 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44575 } }
I20260926 15:19:13.177141 23576 leader_election.cc:302] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f8b515c2a034cbdb9c1d05bdd38c55c; no voters:
I20260926 15:19:13.177718 23576 leader_election.cc:288] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 15:19:13.177834 23580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 3 FOLLOWER]: Leader election won for term 3
I20260926 15:19:13.179080 23580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [term 3 LEADER]: Becoming Leader. State: Replica: 5f8b515c2a034cbdb9c1d05bdd38c55c, State: Running, Role: LEADER
I20260926 15:19:13.179638 23580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44575 } }
I20260926 15:19:13.179814 23576 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:19:13.182485 23581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44575 } } }
I20260926 15:19:13.182884 23581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:13.182477 23582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f8b515c2a034cbdb9c1d05bdd38c55c. Latest consensus state: current_term: 3 leader_uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8b515c2a034cbdb9c1d05bdd38c55c" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 44575 } } }
I20260926 15:19:13.183284 23582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:13.184068 23585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:19:13.199375 23585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:19:13.201221 23585 catalog_manager.cc:1259] Loaded cluster ID: 507c68b71ccf4dc2bd7ec78c4a97dab0
I20260926 15:19:13.201417 23585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:19:13.206235 23585 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:19:13.208141 23585 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c: Loaded TSK: 0
I20260926 15:19:13.208673 23585 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5f8b515c2a034cbdb9c1d05bdd38c55c: Loaded TSK: 1
F20260926 15:19:13.210132 23585 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fccb72a09ad google::LogMessage::Fail() at ??:0
@ 0x7fccb72a388a google::LogMessage::SendToLog() at ??:0
@ 0x7fccb72a04de google::LogMessage::Flush() at ??:0
@ 0x7fccb72a1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fccbb0047d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fccbaff373e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fccccccf1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fcccccce783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fccccd5a2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fccccd5a0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fccccda45e9 std::function<>::operator()() at ??:0
@ 0x7fcccccd2cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fcccccd0c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fccccd58610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fccccd58439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fccb9eec345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fccb9eef91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fccb9eef799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fccb9ecaeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fccbd1a46db start_thread at ??:0
@ 0x7fccb613a61f clone at ??:0
W20260926 15:19:13.479481 23578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:13.479796 23578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:13.480001 23578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:13.487181 23578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:13.487332 23578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:13.487505 23578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:19:13.496338 23578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:46439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=44863
--tserver_master_addrs=127.6.209.254:44575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:13.497572 23578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:13.498890 23578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:13.511201 23623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:13.511341 23626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:13.511204 23624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:13.512028 23578 server_base.cc:1034] running on GCE node
I20260926 15:19:13.512641 23578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:13.513531 23578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:13.514760 23578 hybrid_clock.cc:648] HybridClock initialized: now 1790435953514703 us; error 45 us; skew 500 ppm
I20260926 15:19:13.515041 23578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:19:13 dist-test-slave-j6q7 krb5kdc[22430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435953, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:19:13.518165 23578 init.cc:377] Logged in from keytab as kudu/127.6.209.193@KRBTEST.COM (short username kudu)
I20260926 15:19:13.519562 23578 webserver.cc:460] Webserver started at http://127.6.209.193:44863/ using document root <none> and password file <none>
I20260926 15:19:13.520229 23578 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:13.520361 23578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:13.525632 23578 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:19:13.528314 23633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:13.529598 23578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:19:13.529784 23578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "bab30ea8c2ee4e2fa84ed656a989b777"
format_stamp: "Formatted at 2026-09-26 15:18:28 on dist-test-slave-j6q7"
server_key: "23a13246a18d746cf5642bf2b2c8c0c1"
server_key_iv: "1b5fde8d764dc8eebf53b26fd80d3490"
server_key_version: "encryptionkey@0"
I20260926 15:19:13.530669 23578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:13.547176 23578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:13.550994 23578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:13.551198 23578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:13.551996 23578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:19:13.553568 23640 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260926 15:19:13.558363 23578 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 15:19:13.558509 23578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260926 15:19:13.558646 23578 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 15:19:13.561544 23578 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 15:19:13.561637 23578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.002s
I20260926 15:19:13.561925 23640 tablet_bootstrap.cc:492] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777: Bootstrap starting.
I20260926 15:19:13.569075 23640 log.cc:824] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:13.582474 23640 tablet_bootstrap.cc:492] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:19:13.583258 23640 tablet_bootstrap.cc:492] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777: Bootstrap complete.
I20260926 15:19:13.583796 23640 ts_tablet_manager.cc:1397] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777: Time spent bootstrapping tablet: real 0.022s user 0.020s sys 0.000s
I20260926 15:19:13.593037 23640 raft_consensus.cc:348] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } } peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } }
I20260926 15:19:13.594434 23640 raft_consensus.cc:729] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bab30ea8c2ee4e2fa84ed656a989b777, State: Initialized, Role: FOLLOWER
I20260926 15:19:13.594941 23640 consensus_queue.cc:260] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } } peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } }
I20260926 15:19:13.596319 23640 ts_tablet_manager.cc:1428] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260926 15:19:13.599764 23578 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:46439
I20260926 15:19:13.599833 23748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:46439 every 8 connection(s)
I20260926 15:19:13.600919 23578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:19:13.604646 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:13.603162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.193:53395)
0926 15:19:13.603778 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:13.603981 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":255,"threads_started":1}
W20260926 15:19:13.605269 23749 heartbeater.cc:644] Failed to heartbeat to 127.6.209.254:44575 (0 consecutive failures): Network error: Failed to ping master at 127.6.209.254:44575: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
I20260926 15:19:13.605962 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:13.605711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.193:39333)
0926 15:19:13.605825 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:13.605880 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 15:19:13.606328 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 23578
I20260926 15:19:13.606709 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:13.606397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.193:34047)
0926 15:19:13.606563 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:13.606616 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260926 15:19:13.606935 23749 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 15:19:13.606930 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:34287
--local_ip_for_outbound_sockets=127.6.209.194
--tserver_master_addrs=127.6.209.254:44575
--webserver_port=42025
--webserver_interface=127.6.209.194
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:19:13.900501 23752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:13.900735 23752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:13.900846 23752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:13.907049 23752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:13.907128 23752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:13.907214 23752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:19:13.915189 23752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:34287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=42025
--tserver_master_addrs=127.6.209.254:44575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:13.916078 23752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:13.917097 23752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:13.928515 23758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:13.928632 23760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:13.929035 23757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:13.929316 23752 server_base.cc:1034] running on GCE node
I20260926 15:19:13.929857 23752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:13.930580 23752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:13.931793 23752 hybrid_clock.cc:648] HybridClock initialized: now 1790435953931757 us; error 45 us; skew 500 ppm
I20260926 15:19:13.932061 23752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:19:13 dist-test-slave-j6q7 krb5kdc[22430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435953, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:19:13.935097 23752 init.cc:377] Logged in from keytab as kudu/127.6.209.194@KRBTEST.COM (short username kudu)
I20260926 15:19:13.936367 23752 webserver.cc:460] Webserver started at http://127.6.209.194:42025/ using document root <none> and password file <none>
I20260926 15:19:13.937011 23752 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:13.937141 23752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:13.941812 23752 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 15:19:13.944104 23767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:13.945344 23752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:19:13.945502 23752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "b0f73802c54b46b09e4d723173fc3806"
format_stamp: "Formatted at 2026-09-26 15:18:28 on dist-test-slave-j6q7"
server_key: "5643f667f15ca64af786e09314fd95bd"
server_key_iv: "47359fd8b7326e0f8ce53018dc4cc8f7"
server_key_version: "encryptionkey@0"
I20260926 15:19:13.946472 23752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:13.970808 23752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:13.974033 23752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:13.974262 23752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:13.975037 23752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:19:13.976096 23752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:19:13.976199 23752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:13.976323 23752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:19:13.976379 23752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:14.006371 23752 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:34287
I20260926 15:19:14.006450 23879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:34287 every 8 connection(s)
I20260926 15:19:14.007568 23752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:19:14.011718 23882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:14.010083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.194:54237)
0926 15:19:14.010775 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:14.010995 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":278,"threads_started":1}
W20260926 15:19:14.012293 23880 heartbeater.cc:644] Failed to heartbeat to 127.6.209.254:44575 (0 consecutive failures): Network error: Failed to ping master at 127.6.209.254:44575: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
I20260926 15:19:14.013048 23882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:14.012749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.194:57169)
0926 15:19:14.012895 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:14.012960 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 15:19:14.013756 23882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:14.013473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.194:47935)
0926 15:19:14.013594 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:14.013671 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 15:19:14.014034 23880 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 15:19:14.017100 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 23752
I20260926 15:19:14.017668 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:37809
--local_ip_for_outbound_sockets=127.6.209.195
--tserver_master_addrs=127.6.209.254:44575
--webserver_port=33707
--webserver_interface=127.6.209.195
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:19:14.309563 23883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:14.309789 23883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:14.309964 23883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:14.316716 23883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:14.316797 23883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:14.316934 23883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:19:14.325587 23883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:37809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=33707
--tserver_master_addrs=127.6.209.254:44575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:14.326604 23883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:14.327730 23883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:14.339629 23891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:14.339653 23888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:14.339648 23889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:14.340317 23883 server_base.cc:1034] running on GCE node
I20260926 15:19:14.340929 23883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:14.341809 23883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:14.343016 23883 hybrid_clock.cc:648] HybridClock initialized: now 1790435954342976 us; error 56 us; skew 500 ppm
I20260926 15:19:14.343297 23883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:19:14 dist-test-slave-j6q7 krb5kdc[22430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435954, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:19:14.346040 23883 init.cc:377] Logged in from keytab as kudu/127.6.209.195@KRBTEST.COM (short username kudu)
I20260926 15:19:14.347235 23883 webserver.cc:460] Webserver started at http://127.6.209.195:33707/ using document root <none> and password file <none>
I20260926 15:19:14.347813 23883 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:14.347939 23883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:14.352986 23883 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:19:14.355381 23898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:14.356575 23883 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:19:14.356736 23883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "85783935ae1944c7afbb471f87c8abef"
format_stamp: "Formatted at 2026-09-26 15:18:29 on dist-test-slave-j6q7"
server_key: "f331207c1de312149c3bf1cfba35dc93"
server_key_iv: "b470db8e3f3ece21b33824a04d96bfc1"
server_key_version: "encryptionkey@0"
I20260926 15:19:14.357676 23883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:14.390440 23883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:14.394156 23883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:14.394424 23883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:14.395262 23883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:19:14.396369 23883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:19:14.396492 23883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:14.396605 23883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:19:14.396704 23883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:14.426740 23883 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:37809
I20260926 15:19:14.426816 24010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:37809 every 8 connection(s)
I20260926 15:19:14.427924 23883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:19:14.432472 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:14.430780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.195:37065)
0926 15:19:14.431472 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:14.431678 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":317,"threads_started":1}
W20260926 15:19:14.433202 24011 heartbeater.cc:644] Failed to heartbeat to 127.6.209.254:44575 (0 consecutive failures): Network error: Failed to ping master at 127.6.209.254:44575: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
I20260926 15:19:14.433861 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:14.433579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.195:48479)
0926 15:19:14.433714 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:14.433771 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 15:19:14.434580 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:14.434307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.195:44285)
0926 15:19:14.434425 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:14.434493 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 15:19:14.434801 24011 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 15:19:14.437081 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 23883
I20260926 15:19:14.608018 24015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:14.607432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.193:37221)
0926 15:19:14.607781 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:14.607845 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":168,"threads_started":1}
I20260926 15:19:14.798810 24016 raft_consensus.cc:482] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:19:14.799119 24016 raft_consensus.cc:504] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } } peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } }
I20260926 15:19:14.800745 24016 leader_election.cc:288] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85783935ae1944c7afbb471f87c8abef (127.6.209.195:37809), b0f73802c54b46b09e4d723173fc3806 (127.6.209.194:34287)
Sep 26 15:19:14 dist-test-slave-j6q7 krb5kdc[22430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435953, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for kudu/127.6.209.194@KRBTEST.COM
Sep 26 15:19:14 dist-test-slave-j6q7 krb5kdc[22430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435953, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for kudu/127.6.209.195@KRBTEST.COM
I20260926 15:19:14.820370 24015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:14.801054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:37809 (local address 127.6.209.193:42731)
0926 15:19:14.801382 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:14.801421 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:14.802060 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:14.806161 (+ 4101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:14.806169 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:14.806645 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:19:14.807267 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:14.807285 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:14.808602 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:14.808607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:14.810034 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:14.810044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:14.810884 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:14.810893 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:14.811102 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:14.811817 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:19:14.811831 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:19:14.814012 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:19:14.817322 (+ 3310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:19:14.817326 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:19:14.817330 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:19:14.817673 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:19:14.818024 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:19:14.818028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:19:14.818031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:19:14.818180 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:19:14.818843 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:19:14.818850 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:19:14.819155 (+ 305us) client_negotiation.cc:769] Sending connection context
0926 15:19:14.819413 (+ 258us) client_negotiation.cc:240] Negotiation successful
0926 15:19:14.819933 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":152}
I20260926 15:19:14.820381 24019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:14.801422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:58701 (local address 127.6.209.194:34287)
0926 15:19:14.802845 (+ 1423us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:14.802863 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:14.803347 (+ 484us) server_negotiation.cc:407] Connection header received
0926 15:19:14.805247 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:14.805266 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:14.805729 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:14.806056 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:14.807447 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:14.808423 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:14.810160 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:14.810699 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:14.810876 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:14.813990 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:14.813996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:14.814007 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:14.814035 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:14.817103 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:14.817764 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:14.817773 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:14.817777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:14.817903 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:14.818309 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:14.818313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:14.818316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:14.818717 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:14.818912 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:14.819642 (+ 730us) server_negotiation.cc:299] Negotiation successful
0926 15:19:14.819985 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1219,"thread_start_us":1082,"threads_started":1}
I20260926 15:19:14.820389 24018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:14.801263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:42731 (local address 127.6.209.195:37809)
0926 15:19:14.803209 (+ 1946us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:14.803219 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:14.803254 (+ 35us) server_negotiation.cc:407] Connection header received
0926 15:19:14.805312 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:14.805320 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:14.805729 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:14.806056 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:14.807442 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:14.808453 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:14.810158 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:14.810693 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:14.810876 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:14.814231 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:14.814237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:14.814253 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:14.814285 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:14.817202 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:14.817787 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:14.817799 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:14.817804 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:14.817926 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:14.818286 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:14.818290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:14.818293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:14.818711 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:14.818898 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:14.819640 (+ 742us) server_negotiation.cc:299] Negotiation successful
0926 15:19:14.819997 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1774,"thread_start_us":1598,"threads_started":1}
I20260926 15:19:14.820472 24017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:14.801113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:34287 (local address 127.6.209.193:58701)
0926 15:19:14.803242 (+ 2129us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:14.803260 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:14.803344 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:14.806573 (+ 3229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:14.806576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:14.806977 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:19:14.807267 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:14.807289 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:14.808597 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:14.808600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:14.810035 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:14.810045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:14.810910 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:14.810916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:14.811103 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:14.811764 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:19:14.811786 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:19:14.813753 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:19:14.817224 (+ 3471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:19:14.817231 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:19:14.817242 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:19:14.817631 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:19:14.818005 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:19:14.818009 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:19:14.818012 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:19:14.818206 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:19:14.818847 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:19:14.818852 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:19:14.819157 (+ 305us) client_negotiation.cc:769] Sending connection context
0926 15:19:14.819414 (+ 257us) client_negotiation.cc:240] Negotiation successful
0926 15:19:14.820068 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2041,"mutex_wait_us":418,"spinlock_wait_cycles":20736,"thread_start_us":1882,"threads_started":1}
W20260926 15:19:14.823354 23634 leader_election.cc:341] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0f73802c54b46b09e4d723173fc3806 (127.6.209.194:34287): Not found: Tablet not found: 5aaf2666eb70435fbad886cf6402fcc1
W20260926 15:19:14.823553 23637 leader_election.cc:341] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 85783935ae1944c7afbb471f87c8abef (127.6.209.195:37809): Not found: Tablet not found: 5aaf2666eb70435fbad886cf6402fcc1
I20260926 15:19:14.823655 23637 leader_election.cc:302] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bab30ea8c2ee4e2fa84ed656a989b777; no voters: 85783935ae1944c7afbb471f87c8abef, b0f73802c54b46b09e4d723173fc3806
I20260926 15:19:14.824011 24016 raft_consensus.cc:2726] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 15:19:15.015298 24020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:15.014518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.194:39549)
0926 15:19:15.014944 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:15.015058 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":218,"threads_started":1}
I20260926 15:19:15.435786 24021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:15.435195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.195:44205)
0926 15:19:15.435541 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:15.435629 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":176,"threads_started":1}
I20260926 15:19:15.609560 24022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:15.608987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.193:47745)
0926 15:19:15.609349 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:15.609420 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":188,"threads_started":1}
I20260926 15:19:16.016700 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:16.016136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.194:40637)
0926 15:19:16.016490 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:16.016557 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":194,"threads_started":1}
I20260926 15:19:16.346558 24024 raft_consensus.cc:482] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:19:16.346757 24024 raft_consensus.cc:504] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } } peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } }
I20260926 15:19:16.347325 24024 leader_election.cc:288] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85783935ae1944c7afbb471f87c8abef (127.6.209.195:37809), b0f73802c54b46b09e4d723173fc3806 (127.6.209.194:34287)
W20260926 15:19:16.348140 23637 leader_election.cc:341] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 85783935ae1944c7afbb471f87c8abef (127.6.209.195:37809): Not found: Tablet not found: 5aaf2666eb70435fbad886cf6402fcc1
W20260926 15:19:16.348305 23634 leader_election.cc:341] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0f73802c54b46b09e4d723173fc3806 (127.6.209.194:34287): Not found: Tablet not found: 5aaf2666eb70435fbad886cf6402fcc1
I20260926 15:19:16.348395 23634 leader_election.cc:302] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bab30ea8c2ee4e2fa84ed656a989b777; no voters: 85783935ae1944c7afbb471f87c8abef, b0f73802c54b46b09e4d723173fc3806
I20260926 15:19:16.348587 24024 raft_consensus.cc:2726] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 15:19:16.437165 24025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:16.436518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.195:49777)
0926 15:19:16.436925 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:16.437009 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":189,"threads_started":1}
I20260926 15:19:16.611109 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:16.610312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.193:40729)
0926 15:19:16.610802 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:16.610918 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":200,"threads_started":1}
I20260926 15:19:17.018029 24027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:17.017443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.194:59763)
0926 15:19:17.017824 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:17.017894 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":188,"threads_started":1}
I20260926 15:19:17.438442 24028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:17.437861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.195:50967)
0926 15:19:17.438228 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:17.438310 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":202,"threads_started":1}
I20260926 15:19:17.612586 24029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:17.612013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.193:38807)
0926 15:19:17.612372 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:17.612446 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":192,"threads_started":1}
I20260926 15:19:18.019564 24030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:18.018820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.194:35935)
0926 15:19:18.019286 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:18.019358 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":261,"threads_started":1}
I20260926 15:19:18.440088 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:18.439414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.195:53139)
0926 15:19:18.439816 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:18.439912 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":193,"threads_started":1}
I20260926 15:19:18.614307 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:18.613553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.193:35515)
0926 15:19:18.614071 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:18.614148 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":268,"threads_started":1}
I20260926 15:19:19.021078 24033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:19.020405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.194:37249)
0926 15:19:19.020839 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:19.020918 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":211,"threads_started":1}
I20260926 15:19:19.427415 24034 raft_consensus.cc:482] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:19:19.427596 24034 raft_consensus.cc:504] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } } peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } }
I20260926 15:19:19.428191 24034 leader_election.cc:288] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85783935ae1944c7afbb471f87c8abef (127.6.209.195:37809), b0f73802c54b46b09e4d723173fc3806 (127.6.209.194:34287)
W20260926 15:19:19.429134 23637 leader_election.cc:341] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 85783935ae1944c7afbb471f87c8abef (127.6.209.195:37809): Not found: Tablet not found: 5aaf2666eb70435fbad886cf6402fcc1
W20260926 15:19:19.429266 23634 leader_election.cc:341] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0f73802c54b46b09e4d723173fc3806 (127.6.209.194:34287): Not found: Tablet not found: 5aaf2666eb70435fbad886cf6402fcc1
I20260926 15:19:19.429433 23634 leader_election.cc:302] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bab30ea8c2ee4e2fa84ed656a989b777; no voters: 85783935ae1944c7afbb471f87c8abef, b0f73802c54b46b09e4d723173fc3806
I20260926 15:19:19.429662 24034 raft_consensus.cc:2726] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 15:19:19.441521 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:19.440915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.195:39647)
0926 15:19:19.441328 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:19.441398 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":215,"threads_started":1}
I20260926 15:19:19.615763 24036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:19.615181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.193:51339)
0926 15:19:19.615537 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:19.615610 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":190,"threads_started":1}
I20260926 15:19:20.022516 24037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:20.021838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.194:46057)
0926 15:19:20.022284 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:20.022359 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":240,"threads_started":1}
I20260926 15:19:20.442879 24038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:20.442241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.195:35987)
0926 15:19:20.442644 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:20.442718 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":221,"threads_started":1}
I20260926 15:19:20.617264 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:20.616627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.193:45841)
0926 15:19:20.617041 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:20.617114 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":236,"threads_started":1}
I20260926 15:19:21.024019 24040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:21.023354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.194:57599)
0926 15:19:21.023795 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:21.023868 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":204,"threads_started":1}
I20260926 15:19:21.444490 24041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:21.443806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.195:41641)
0926 15:19:21.444241 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:21.444311 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":192,"threads_started":1}
I20260926 15:19:21.618621 24042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:21.618034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.193:38057)
0926 15:19:21.618422 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:21.618491 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":221,"threads_started":1}
I20260926 15:19:21.699967 24043 raft_consensus.cc:482] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:19:21.700233 24043 raft_consensus.cc:504] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } } peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } }
I20260926 15:19:21.701000 24043 leader_election.cc:288] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85783935ae1944c7afbb471f87c8abef (127.6.209.195:37809), b0f73802c54b46b09e4d723173fc3806 (127.6.209.194:34287)
W20260926 15:19:21.701788 23637 leader_election.cc:341] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 85783935ae1944c7afbb471f87c8abef (127.6.209.195:37809): Not found: Tablet not found: 5aaf2666eb70435fbad886cf6402fcc1
W20260926 15:19:21.701951 23634 leader_election.cc:341] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0f73802c54b46b09e4d723173fc3806 (127.6.209.194:34287): Not found: Tablet not found: 5aaf2666eb70435fbad886cf6402fcc1
I20260926 15:19:21.702139 23634 leader_election.cc:302] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bab30ea8c2ee4e2fa84ed656a989b777; no voters: 85783935ae1944c7afbb471f87c8abef, b0f73802c54b46b09e4d723173fc3806
I20260926 15:19:21.702387 24043 raft_consensus.cc:2726] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 15:19:22.025499 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:22.024959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.194:53145)
0926 15:19:22.025286 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:22.025352 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":163,"threads_started":1}
I20260926 15:19:22.445717 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:22.445185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.195:33799)
0926 15:19:22.445510 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:22.445581 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":172,"threads_started":1}
I20260926 15:19:22.620217 24046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:22.619507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.193:54609)
0926 15:19:22.619977 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:22.620058 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":230,"threads_started":1}
I20260926 15:19:23.026908 24047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:23.026327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.194:35647)
0926 15:19:23.026704 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:23.026772 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":178,"threads_started":1}
I20260926 15:19:23.315011 24048 raft_consensus.cc:482] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:19:23.315210 24048 raft_consensus.cc:504] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85783935ae1944c7afbb471f87c8abef" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 37809 } } peers { permanent_uuid: "bab30ea8c2ee4e2fa84ed656a989b777" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 46439 } } peers { permanent_uuid: "b0f73802c54b46b09e4d723173fc3806" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 34287 } }
I20260926 15:19:23.315876 24048 leader_election.cc:288] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85783935ae1944c7afbb471f87c8abef (127.6.209.195:37809), b0f73802c54b46b09e4d723173fc3806 (127.6.209.194:34287)
W20260926 15:19:23.316684 23637 leader_election.cc:341] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 85783935ae1944c7afbb471f87c8abef (127.6.209.195:37809): Not found: Tablet not found: 5aaf2666eb70435fbad886cf6402fcc1
W20260926 15:19:23.316871 23634 leader_election.cc:341] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0f73802c54b46b09e4d723173fc3806 (127.6.209.194:34287): Not found: Tablet not found: 5aaf2666eb70435fbad886cf6402fcc1
I20260926 15:19:23.317021 23634 leader_election.cc:302] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bab30ea8c2ee4e2fa84ed656a989b777; no voters: 85783935ae1944c7afbb471f87c8abef, b0f73802c54b46b09e4d723173fc3806
I20260926 15:19:23.317236 24048 raft_consensus.cc:2726] T 5aaf2666eb70435fbad886cf6402fcc1 P bab30ea8c2ee4e2fa84ed656a989b777 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 15:19:23.447135 24049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:23.446519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.195:56553)
0926 15:19:23.446922 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:23.446997 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":190,"threads_started":1}
I20260926 15:19:23.621737 24050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:23.621120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.193:41373)
0926 15:19:23.621502 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:23.621588 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":199,"threads_started":1}
I20260926 15:19:24.028223 24051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:24.027621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.194:59377)
0926 15:19:24.027969 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:24.028062 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":188,"threads_started":1}
I20260926 15:19:24.448545 24052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:24.447934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.195:50257)
0926 15:19:24.448330 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:24.448398 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":197,"threads_started":1}
I20260926 15:19:24.623133 24053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:24.622485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.193:37235)
0926 15:19:24.622882 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:24.622974 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":224,"threads_started":1}
I20260926 15:19:25.029740 24054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:25.029076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.194:41141)
0926 15:19:25.029505 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:25.029585 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":228,"threads_started":1}
I20260926 15:19:25.450065 24055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:25.449420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.195:37561)
0926 15:19:25.449804 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:25.449932 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":182,"threads_started":1}
I20260926 15:19:25.624552 24056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:25.623961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.193:52383)
0926 15:19:25.624337 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:25.624408 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":199,"threads_started":1}
I20260926 15:19:26.031203 24057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:26.030550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.194:52783)
0926 15:19:26.030964 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:26.031059 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":233,"threads_started":1}
I20260926 15:19:26.451433 24058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:26.450858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.195:37047)
0926 15:19:26.451232 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:26.451300 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":207,"threads_started":1}
I20260926 15:19:26.625989 24059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:26.625401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.193:49501)
0926 15:19:26.625771 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:26.625846 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":193,"threads_started":1}
I20260926 15:19:27.032927 24060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:27.032234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.194:49905)
0926 15:19:27.032617 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:27.032708 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":198,"threads_started":1}
I20260926 15:19:27.452862 24061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:27.452235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.195:43927)
0926 15:19:27.452619 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:27.452698 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":206,"threads_started":1}
I20260926 15:19:27.627501 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:27.626904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.193:35331)
0926 15:19:27.627272 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:27.627340 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":193,"threads_started":1}
I20260926 15:19:28.034333 24063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:28.033644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.194:53419)
0926 15:19:28.034069 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:28.034145 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":216,"threads_started":1}
I20260926 15:19:28.454062 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:28.453502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.195:58793)
0926 15:19:28.453842 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:28.453909 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":183,"threads_started":1}
I20260926 15:19:28.629122 24065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:28.628526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.193:55145)
0926 15:19:28.628906 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:28.628983 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":175,"threads_started":1}
I20260926 15:19:29.035866 24066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:29.035269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.194:58737)
0926 15:19:29.035640 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:29.035726 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":191,"threads_started":1}
I20260926 15:19:29.439685 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 23578
I20260926 15:19:29.452349 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:29.440643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60032 (local address 127.6.209.193:46439)
0926 15:19:29.441192 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:29.441202 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:29.441236 (+ 34us) server_negotiation.cc:407] Connection header received
0926 15:19:29.441342 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:29.441351 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:29.441643 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:29.441813 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:29.443318 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:29.444153 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:29.445219 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:29.445724 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:29.445807 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:29.447841 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:29.447851 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:29.447863 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:29.447893 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:29.450266 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:29.450859 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:29.450866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:29.450868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:29.451004 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:29.451313 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:29.451316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:29.451318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:29.451615 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:29.451805 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:29.451969 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 15:19:29.452128 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":254,"threads_started":1}
I20260926 15:19:29.453538 23744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:29.455626 24069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:29.454957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:44575 (local address 127.6.209.195:33061)
0926 15:19:29.455357 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:29.455454 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.254:44575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":219,"threads_started":1}
I20260926 15:19:29.531721 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 23578
I20260926 15:19:29.547665 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 23752
I20260926 15:19:29.559079 24070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:29.548318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33780 (local address 127.6.209.194:34287)
0926 15:19:29.548667 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:29.548670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:29.548685 (+ 15us) server_negotiation.cc:407] Connection header received
0926 15:19:29.548740 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:29.548744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:29.548793 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:29.548919 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:29.550115 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:29.550928 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:29.552040 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:29.552436 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:29.552516 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:29.554683 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:29.554688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:29.554691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:29.554723 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:29.557132 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:29.557631 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:29.557634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:29.557636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:29.557750 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:29.558015 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:29.558018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:29.558020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:29.558239 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:29.558412 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:29.558598 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 15:19:29.558743 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":182,"threads_started":1}
I20260926 15:19:29.559757 23875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:29.634779 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 23752
I20260926 15:19:29.651038 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 23883
I20260926 15:19:29.663236 24072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:29.651865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58452 (local address 127.6.209.195:37809)
0926 15:19:29.652401 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:29.652406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:29.652421 (+ 15us) server_negotiation.cc:407] Connection header received
0926 15:19:29.652469 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:29.652472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:29.652526 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:29.652609 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:29.653854 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:29.654621 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:29.655657 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:29.656032 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:29.656113 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:29.658319 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:29.658323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:29.658326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:29.658353 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:29.661038 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:29.661656 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:29.661660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:29.661662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:29.661795 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:29.662091 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:29.662094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:29.662096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:29.662305 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:29.662541 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:29.662702 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 15:19:29.662849 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":306,"threads_started":1}
I20260926 15:19:29.663862 24006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:29.741967 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 23883
2026-09-26T15:19:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (63381 ms)
[----------] 30 tests from SecurityITest (275283 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:19:29 dist-test-slave-j6q7 krb5kdc[24076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:19:29 dist-test-slave-j6q7 krb5kdc[24076](info): set up 2 sockets
Sep 26 15:19:29 dist-test-slave-j6q7 krb5kdc[24076](info): commencing operation
krb5kdc: starting...
W20260926 15:19:30.744779 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:19:30 dist-test-slave-j6q7 krb5kdc[24076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:19:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:19:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:19:30.947389 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:36125
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:40083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:36125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:19:31.238140 24089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:31.238425 24089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:31.238560 24089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:31.245067 24089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:19:31.245182 24089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:31.245251 24089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:31.245324 24089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:19:31.245383 24089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:19:31.253916 24089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:40083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:36125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:36125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:31.255075 24089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:31.256280 24089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:31.267096 24097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:31.267130 24094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:31.267158 24095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:31.268028 24089 server_base.cc:1034] running on GCE node
I20260926 15:19:31.268644 24089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:31.269788 24089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:31.270972 24089 hybrid_clock.cc:648] HybridClock initialized: now 1790435971270920 us; error 40 us; skew 500 ppm
I20260926 15:19:31.271289 24089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:19:31 dist-test-slave-j6q7 krb5kdc[24076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435971, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:19:31.274133 24089 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:19:31.275315 24089 webserver.cc:460] Webserver started at http://127.6.209.254:44673/ using document root <none> and password file <none>
I20260926 15:19:31.275831 24089 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:31.275915 24089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:31.276118 24089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:31.278563 24089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "7cc5c7d3f74c4126a85558022db8d8a3"
format_stamp: "Formatted at 2026-09-26 15:19:31 on dist-test-slave-j6q7"
server_key: "55359ca76238ad17eef7b895b4196f78"
server_key_iv: "154a0492d5b68bbf835592b7f225b9c3"
server_key_version: "encryptionkey@0"
I20260926 15:19:31.279196 24089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "7cc5c7d3f74c4126a85558022db8d8a3"
format_stamp: "Formatted at 2026-09-26 15:19:31 on dist-test-slave-j6q7"
server_key: "55359ca76238ad17eef7b895b4196f78"
server_key_iv: "154a0492d5b68bbf835592b7f225b9c3"
server_key_version: "encryptionkey@0"
I20260926 15:19:31.284253 24089 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 15:19:31.287317 24104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:31.288434 24089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:19:31.288604 24089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "7cc5c7d3f74c4126a85558022db8d8a3"
format_stamp: "Formatted at 2026-09-26 15:19:31 on dist-test-slave-j6q7"
server_key: "55359ca76238ad17eef7b895b4196f78"
server_key_iv: "154a0492d5b68bbf835592b7f225b9c3"
server_key_version: "encryptionkey@0"
I20260926 15:19:31.288782 24089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:31.301368 24089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:31.305363 24089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:31.305630 24089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:31.321055 24089 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:36125
I20260926 15:19:31.321074 24155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:36125 every 8 connection(s)
I20260926 15:19:31.322234 24089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:19:31.324798 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 24089
I20260926 15:19:31.325058 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:19:31.325692 24156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:31.325780 6983 external_mini_cluster.cc:1428] Setting key 7f1fb68d4812873dc4dd92bf9e334552
I20260926 15:19:31.338452 24156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cc5c7d3f74c4126a85558022db8d8a3: Bootstrap starting.
Sep 26 15:19:31 dist-test-slave-j6q7 krb5kdc[24076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:19:31.341401 24156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cc5c7d3f74c4126a85558022db8d8a3: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:31.342315 24156 log.cc:824] T 00000000000000000000000000000000 P 7cc5c7d3f74c4126a85558022db8d8a3: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:31.344563 24156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cc5c7d3f74c4126a85558022db8d8a3: No bootstrap required, opened a new log
I20260926 15:19:31.345912 24159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:31.327417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.6.209.254:36125)
0926 15:19:31.328059 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:31.328069 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:31.328109 (+ 40us) server_negotiation.cc:407] Connection header received
0926 15:19:31.329790 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:31.329808 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:31.330109 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:31.330430 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:31.332105 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:31.333293 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:31.334867 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:31.335412 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:31.335523 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:31.339704 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:31.339710 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:31.339724 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:31.339756 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:31.342619 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:31.343174 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:31.343180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:31.343185 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:31.343311 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:31.343642 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:31.343648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:31.343651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:31.344102 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:31.344382 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:31.344780 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 15:19:31.345165 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":296,"threads_started":1}
I20260926 15:19:31.351825 24156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cc5c7d3f74c4126a85558022db8d8a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cc5c7d3f74c4126a85558022db8d8a3" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 36125 } }
I20260926 15:19:31.352191 24156 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cc5c7d3f74c4126a85558022db8d8a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:31.352298 24156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cc5c7d3f74c4126a85558022db8d8a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cc5c7d3f74c4126a85558022db8d8a3, State: Initialized, Role: FOLLOWER
I20260926 15:19:31.352898 24156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cc5c7d3f74c4126a85558022db8d8a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cc5c7d3f74c4126a85558022db8d8a3" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 36125 } }
I20260926 15:19:31.353101 24156 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cc5c7d3f74c4126a85558022db8d8a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:19:31.353183 24156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cc5c7d3f74c4126a85558022db8d8a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:19:31.353286 24156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cc5c7d3f74c4126a85558022db8d8a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:31.354879 24156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cc5c7d3f74c4126a85558022db8d8a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cc5c7d3f74c4126a85558022db8d8a3" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 36125 } }
I20260926 15:19:31.355230 24156 leader_election.cc:302] T 00000000000000000000000000000000 P 7cc5c7d3f74c4126a85558022db8d8a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cc5c7d3f74c4126a85558022db8d8a3; no voters:
I20260926 15:19:31.355675 24156 leader_election.cc:288] T 00000000000000000000000000000000 P 7cc5c7d3f74c4126a85558022db8d8a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:19:31.355748 24161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cc5c7d3f74c4126a85558022db8d8a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:19:31.356382 24161 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cc5c7d3f74c4126a85558022db8d8a3 [term 1 LEADER]: Becoming Leader. State: Replica: 7cc5c7d3f74c4126a85558022db8d8a3, State: Running, Role: LEADER
I20260926 15:19:31.356916 24161 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cc5c7d3f74c4126a85558022db8d8a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cc5c7d3f74c4126a85558022db8d8a3" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 36125 } }
I20260926 15:19:31.357460 24156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cc5c7d3f74c4126a85558022db8d8a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:19:31.359112 24162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cc5c7d3f74c4126a85558022db8d8a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cc5c7d3f74c4126a85558022db8d8a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cc5c7d3f74c4126a85558022db8d8a3" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 36125 } } }
I20260926 15:19:31.359422 24162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cc5c7d3f74c4126a85558022db8d8a3 [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:31.361056 24163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cc5c7d3f74c4126a85558022db8d8a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cc5c7d3f74c4126a85558022db8d8a3. Latest consensus state: current_term: 1 leader_uuid: "7cc5c7d3f74c4126a85558022db8d8a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cc5c7d3f74c4126a85558022db8d8a3" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 36125 } } }
I20260926 15:19:31.361419 24163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cc5c7d3f74c4126a85558022db8d8a3 [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:31.361963 24168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:19:31.365868 24168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:19:31.373230 24168 catalog_manager.cc:1347] Generated new cluster ID: 910f3563ce21416192cfe6bf57482a2d
I20260926 15:19:31.373323 24168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:19:31.381322 24168 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:19:31.382310 24168 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:19:31.390697 24168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cc5c7d3f74c4126a85558022db8d8a3: Generated new TSK 0
I20260926 15:19:31.391505 24168 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
I20260926 15:19:31.445169 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:36125
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:40083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:19:31.828981 24182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:31.829231 24182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:31.829371 24182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:31.836050 24182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:31.836176 24182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:31.836326 24182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:19:31.844722 24182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:40083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:36125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:31.845692 24182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:31.846871 24182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:31.861176 24188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:31.861238 24190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:31.861783 24187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:31.862165 24182 server_base.cc:1034] running on GCE node
I20260926 15:19:31.862818 24182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:31.863628 24182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:31.864861 24182 hybrid_clock.cc:648] HybridClock initialized: now 1790435971864776 us; error 51 us; skew 500 ppm
I20260926 15:19:31.865134 24182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:19:31 dist-test-slave-j6q7 krb5kdc[24076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435971, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:19:31.873173 24182 init.cc:377] Logged in from keytab as kudu/127.6.209.193@KRBTEST.COM (short username kudu)
I20260926 15:19:31.874505 24182 webserver.cc:460] Webserver started at http://127.6.209.193:39293/ using document root <none> and password file <none>
I20260926 15:19:31.875082 24182 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:31.875213 24182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:31.875480 24182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:31.877887 24182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "6d896f0a3e6543588360a9c614cdd446"
format_stamp: "Formatted at 2026-09-26 15:19:31 on dist-test-slave-j6q7"
server_key: "2faab021b26bc707af8bea828c74653b"
server_key_iv: "eef378356c30440da3f8a01c30b7a18b"
server_key_version: "encryptionkey@0"
I20260926 15:19:31.878672 24182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "6d896f0a3e6543588360a9c614cdd446"
format_stamp: "Formatted at 2026-09-26 15:19:31 on dist-test-slave-j6q7"
server_key: "2faab021b26bc707af8bea828c74653b"
server_key_iv: "eef378356c30440da3f8a01c30b7a18b"
server_key_version: "encryptionkey@0"
I20260926 15:19:31.883473 24182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 15:19:31.886579 24197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:31.887633 24182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 15:19:31.887787 24182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "6d896f0a3e6543588360a9c614cdd446"
format_stamp: "Formatted at 2026-09-26 15:19:31 on dist-test-slave-j6q7"
server_key: "2faab021b26bc707af8bea828c74653b"
server_key_iv: "eef378356c30440da3f8a01c30b7a18b"
server_key_version: "encryptionkey@0"
I20260926 15:19:31.887966 24182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:31.897738 24182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:31.900981 24182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:31.901240 24182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:31.902061 24182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:19:31.903138 24182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:19:31.903234 24182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:31.903362 24182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:19:31.903426 24182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:31.933246 24182 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:44589
I20260926 15:19:31.933306 24309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:44589 every 8 connection(s)
I20260926 15:19:31.934283 24182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:19:31.944142 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 24182
I20260926 15:19:31.944343 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:19:31.945119 6983 external_mini_cluster.cc:1428] Setting key 05809a0b9841ed2d85a1c0a8a65e4f11
Sep 26 15:19:31 dist-test-slave-j6q7 krb5kdc[24076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435971, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:19:31.951582 24312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:31.936264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:50419 (local address 127.6.209.254:36125)
0926 15:19:31.936624 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:31.936628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:31.937314 (+ 686us) server_negotiation.cc:407] Connection header received
0926 15:19:31.938828 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:31.938833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:31.938907 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:31.939011 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:31.940270 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:31.941057 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:31.942432 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:31.942769 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:31.942881 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:31.946352 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:31.946358 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:31.946362 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:31.946393 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:31.948735 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:31.949538 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:31.949542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:31.949544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:31.949657 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:31.950063 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:31.950065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:31.950067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:31.950266 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:31.950381 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:31.951189 (+ 808us) server_negotiation.cc:299] Negotiation successful
0926 15:19:31.951341 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":195,"threads_started":1}
I20260926 15:19:31.952849 24313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:31.936569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:36125 (local address 127.6.209.193:50419)
0926 15:19:31.937180 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:31.937218 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:31.938596 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:31.939200 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:31.939208 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:31.939593 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:19:31.940103 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:31.940118 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:31.941207 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:31.941212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:31.942281 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:31.942294 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:31.942964 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:31.942971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:31.943164 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:31.943694 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:19:31.943739 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:19:31.946161 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:19:31.948887 (+ 2726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:19:31.948898 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:19:31.948913 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:19:31.949441 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:19:31.949758 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:19:31.949764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:19:31.949769 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:19:31.949969 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:19:31.950405 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:19:31.950415 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:19:31.950826 (+ 411us) client_negotiation.cc:769] Sending connection context
0926 15:19:31.951089 (+ 263us) client_negotiation.cc:240] Negotiation successful
0926 15:19:31.951585 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":282,"threads_started":1}
I20260926 15:19:31.954129 24310 heartbeater.cc:347] Connected to a master server at 127.6.209.254:36125
I20260926 15:19:31.954444 24310 heartbeater.cc:464] Registering TS with master...
I20260926 15:19:31.955153 24310 heartbeater.cc:511] Master 127.6.209.254:36125 requested a full tablet report, sending...
I20260926 15:19:31.957121 24121 ts_manager.cc:194] Registered new tserver with Master: 6d896f0a3e6543588360a9c614cdd446 (127.6.209.193:44589)
I20260926 15:19:31.958473 24121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.193@KRBTEST.COM'} at 127.6.209.193:50419
WARNING: no policy specified for kudu/127.6.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
I20260926 15:19:31.995482 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:36125
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:40083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:19:32.278671 24316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:32.278882 24316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:32.278954 24316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:32.285670 24316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:32.285753 24316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:32.285845 24316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:19:32.294054 24316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:40083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:36125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:32.295055 24316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:32.296074 24316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:32.307461 24322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:32.307524 24321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:32.307493 24324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:32.308226 24316 server_base.cc:1034] running on GCE node
I20260926 15:19:32.308677 24316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:32.309537 24316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:32.310722 24316 hybrid_clock.cc:648] HybridClock initialized: now 1790435972310666 us; error 45 us; skew 500 ppm
I20260926 15:19:32.311002 24316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:19:32 dist-test-slave-j6q7 krb5kdc[24076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435972, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:19:32.314342 24316 init.cc:377] Logged in from keytab as kudu/127.6.209.194@KRBTEST.COM (short username kudu)
I20260926 15:19:32.315701 24316 webserver.cc:460] Webserver started at http://127.6.209.194:33071/ using document root <none> and password file <none>
I20260926 15:19:32.316326 24316 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:32.316502 24316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:32.316869 24316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:32.319635 24316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "09917bcb37ff4f358aeeece71e868bcb"
format_stamp: "Formatted at 2026-09-26 15:19:32 on dist-test-slave-j6q7"
server_key: "9909a43c19eeaae1f2e21b8446adc00c"
server_key_iv: "4b569f6dbb40d865ffa589c49fc168cb"
server_key_version: "encryptionkey@0"
I20260926 15:19:32.320469 24316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "09917bcb37ff4f358aeeece71e868bcb"
format_stamp: "Formatted at 2026-09-26 15:19:32 on dist-test-slave-j6q7"
server_key: "9909a43c19eeaae1f2e21b8446adc00c"
server_key_iv: "4b569f6dbb40d865ffa589c49fc168cb"
server_key_version: "encryptionkey@0"
I20260926 15:19:32.325600 24316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 15:19:32.328760 24331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:32.329902 24316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260926 15:19:32.330065 24316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "09917bcb37ff4f358aeeece71e868bcb"
format_stamp: "Formatted at 2026-09-26 15:19:32 on dist-test-slave-j6q7"
server_key: "9909a43c19eeaae1f2e21b8446adc00c"
server_key_iv: "4b569f6dbb40d865ffa589c49fc168cb"
server_key_version: "encryptionkey@0"
I20260926 15:19:32.330242 24316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:32.341768 24316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:32.345252 24316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:32.345513 24316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:32.346326 24316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:19:32.347435 24316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:19:32.347538 24316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:32.347677 24316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:19:32.347752 24316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:32.378711 24316 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:35653
I20260926 15:19:32.378750 24443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:35653 every 8 connection(s)
I20260926 15:19:32.379817 24316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:19:32.383420 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 24316
I20260926 15:19:32.383622 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:19:32.384104 6983 external_mini_cluster.cc:1428] Setting key b3238e1633c480cbd8c831ae6c87ea26
Sep 26 15:19:32 dist-test-slave-j6q7 krb5kdc[24076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435972, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:19:32.398022 24312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:32.382189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:49955 (local address 127.6.209.254:36125)
0926 15:19:32.382338 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:32.382341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:32.383295 (+ 954us) server_negotiation.cc:407] Connection header received
0926 15:19:32.385169 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:32.385174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:32.385257 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:32.385354 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:32.386676 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:32.387294 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:32.388613 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:32.389189 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:32.389300 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:32.393054 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:32.393059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:32.393064 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:32.393095 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:32.395363 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:32.396113 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:32.396116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:32.396118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:32.396237 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:32.396639 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:32.396642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:32.396643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:32.396909 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:32.397013 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:32.397653 (+ 640us) server_negotiation.cc:299] Negotiation successful
0926 15:19:32.397813 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 15:19:32.399258 24446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:32.382556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:36125 (local address 127.6.209.194:49955)
0926 15:19:32.383163 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:32.383200 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:32.384950 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:32.385540 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:32.385546 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:32.386086 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:19:32.386533 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:32.386546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:32.387428 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:32.387432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:32.388479 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:32.388489 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:32.389445 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:32.389455 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:32.389777 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:32.390454 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:19:32.390491 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:19:32.392869 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:19:32.395488 (+ 2619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:19:32.395497 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:19:32.395511 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:19:32.396010 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:19:32.396336 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:19:32.396342 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:19:32.396348 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:19:32.396546 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:19:32.397018 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:19:32.397026 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:19:32.397360 (+ 334us) client_negotiation.cc:769] Sending connection context
0926 15:19:32.397597 (+ 237us) client_negotiation.cc:240] Negotiation successful
0926 15:19:32.398063 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":253,"threads_started":1}
I20260926 15:19:32.400550 24444 heartbeater.cc:347] Connected to a master server at 127.6.209.254:36125
I20260926 15:19:32.400871 24444 heartbeater.cc:464] Registering TS with master...
I20260926 15:19:32.401510 24444 heartbeater.cc:511] Master 127.6.209.254:36125 requested a full tablet report, sending...
I20260926 15:19:32.402769 24121 ts_manager.cc:194] Registered new tserver with Master: 09917bcb37ff4f358aeeece71e868bcb (127.6.209.194:35653)
I20260926 15:19:32.403456 24121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.194@KRBTEST.COM'} at 127.6.209.194:49955
WARNING: no policy specified for kudu/127.6.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
I20260926 15:19:32.432140 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:36125
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:40083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:19:32.724684 24449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:32.725095 24449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:32.725231 24449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:32.732115 24449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:32.732190 24449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:32.732283 24449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:19:32.740964 24449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:40083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:36125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:32.741961 24449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:32.743027 24449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:32.754614 24457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:32.754823 24454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:32.754669 24455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:32.755275 24449 server_base.cc:1034] running on GCE node
I20260926 15:19:32.755839 24449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:32.756680 24449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:32.757887 24449 hybrid_clock.cc:648] HybridClock initialized: now 1790435972757847 us; error 51 us; skew 500 ppm
I20260926 15:19:32.758201 24449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:19:32 dist-test-slave-j6q7 krb5kdc[24076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435972, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:19:32.761060 24449 init.cc:377] Logged in from keytab as kudu/127.6.209.195@KRBTEST.COM (short username kudu)
I20260926 15:19:32.762301 24449 webserver.cc:460] Webserver started at http://127.6.209.195:36917/ using document root <none> and password file <none>
I20260926 15:19:32.762852 24449 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:32.762980 24449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:32.763243 24449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:32.765722 24449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "041e36c1c57548b1863f99be2acbba9b"
format_stamp: "Formatted at 2026-09-26 15:19:32 on dist-test-slave-j6q7"
server_key: "9838bc4ce49d758b5656c6c72bd5b60f"
server_key_iv: "9bd02a2f08f85ca6a4d82ea34dded21d"
server_key_version: "encryptionkey@0"
I20260926 15:19:32.766451 24449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "041e36c1c57548b1863f99be2acbba9b"
format_stamp: "Formatted at 2026-09-26 15:19:32 on dist-test-slave-j6q7"
server_key: "9838bc4ce49d758b5656c6c72bd5b60f"
server_key_iv: "9bd02a2f08f85ca6a4d82ea34dded21d"
server_key_version: "encryptionkey@0"
I20260926 15:19:32.770848 24449 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 15:19:32.773919 24464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:32.775031 24449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:19:32.775192 24449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "041e36c1c57548b1863f99be2acbba9b"
format_stamp: "Formatted at 2026-09-26 15:19:32 on dist-test-slave-j6q7"
server_key: "9838bc4ce49d758b5656c6c72bd5b60f"
server_key_iv: "9bd02a2f08f85ca6a4d82ea34dded21d"
server_key_version: "encryptionkey@0"
I20260926 15:19:32.775367 24449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:32.795115 24449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:32.798790 24449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:32.798990 24449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:32.799712 24449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:19:32.800937 24449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:19:32.801007 24449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:32.801083 24449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:19:32.801126 24449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:32.830286 24449 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:35357
I20260926 15:19:32.830350 24576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:35357 every 8 connection(s)
I20260926 15:19:32.831409 24449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:19:32.839463 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 24449
I20260926 15:19:32.839612 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:19:32.840149 6983 external_mini_cluster.cc:1428] Setting key b2129666ceb75fa17c7ceced01ff9c25
Sep 26 15:19:32 dist-test-slave-j6q7 krb5kdc[24076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435972, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:19:32.848037 24312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:32.833729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:37963 (local address 127.6.209.254:36125)
0926 15:19:32.833905 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:32.833910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:32.834917 (+ 1007us) server_negotiation.cc:407] Connection header received
0926 15:19:32.836347 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:32.836351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:32.836416 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:32.836508 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:32.837728 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:32.838375 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:32.839895 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:32.840315 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:32.840405 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:32.843952 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:32.843955 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:32.843958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:32.843985 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:32.846023 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:32.846557 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:32.846560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:32.846562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:32.846663 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:32.846953 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:32.846955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:32.846957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:32.847159 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:32.847257 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:32.847712 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 15:19:32.847835 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 15:19:32.848717 24579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:32.834143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:36125 (local address 127.6.209.195:37963)
0926 15:19:32.834776 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:32.834806 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:32.836141 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:32.836688 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:32.836694 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:32.837145 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:19:32.837598 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:32.837612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:32.838475 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:32.838478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:32.839747 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:32.839759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:32.840553 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:32.840561 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:32.840866 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:32.841564 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:19:32.841605 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:19:32.843807 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:19:32.846129 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:19:32.846140 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:19:32.846148 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:19:32.846475 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:19:32.846738 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:19:32.846741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:19:32.846743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:19:32.846877 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:19:32.847248 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:19:32.847252 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:19:32.847464 (+ 212us) client_negotiation.cc:769] Sending connection context
0926 15:19:32.847641 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 15:19:32.847910 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":240,"threads_started":1}
I20260926 15:19:32.849767 24577 heartbeater.cc:347] Connected to a master server at 127.6.209.254:36125
I20260926 15:19:32.850000 24577 heartbeater.cc:464] Registering TS with master...
I20260926 15:19:32.850566 24577 heartbeater.cc:511] Master 127.6.209.254:36125 requested a full tablet report, sending...
I20260926 15:19:32.851804 24121 ts_manager.cc:194] Registered new tserver with Master: 041e36c1c57548b1863f99be2acbba9b (127.6.209.195:35357)
I20260926 15:19:32.852548 24121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.195@KRBTEST.COM'} at 127.6.209.195:37963
I20260926 15:19:32.854988 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:19:32.869577 24312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:32.858984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41538 (local address 127.6.209.254:36125)
0926 15:19:32.859095 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:32.859113 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:32.859459 (+ 346us) server_negotiation.cc:407] Connection header received
0926 15:19:32.859634 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:32.859638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:32.859700 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:32.859788 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:32.861177 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:32.861883 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:32.862988 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:32.863336 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:32.863445 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:32.865403 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:32.865408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:32.865412 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:32.865445 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:32.867425 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:32.867970 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:32.867976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:32.867978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:32.868083 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:32.868395 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:32.868400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:32.868402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:32.868644 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:32.868879 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:32.869143 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 15:19:32.869322 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 15:19:32.872611 24121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:19:32.878528 24121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:19:32.908700 24589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:32.888436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:35357 (local address 127.0.0.1:51294)
0926 15:19:32.889086 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:32.889123 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:32.889261 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:32.902001 (+ 12740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:32.902005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:32.902024 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:32.902478 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:32.902489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:32.904135 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:32.904141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:32.906421 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:32.906435 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:32.907715 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:32.907720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:32.907927 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:32.907976 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 15:19:32.908097 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 15:19:32.908276 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":350,"thread_start_us":301,"threads_started":1}
I20260926 15:19:32.908886 24591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:32.889054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:35653 (local address 127.0.0.1:48930)
0926 15:19:32.899342 (+ 10288us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:32.899363 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:32.899482 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:32.901390 (+ 1908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:32.901397 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:32.901428 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:32.901778 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:32.901790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:32.903615 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:32.903620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:32.906348 (+ 2728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:32.906361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:32.908106 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:32.908112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:32.908268 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:32.908292 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 15:19:32.908370 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 15:19:32.908481 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10121,"mutex_wait_us":261,"thread_start_us":9970,"threads_started":1}
I20260926 15:19:32.909243 24594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:32.890492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43512 (local address 127.6.209.193:44589)
0926 15:19:32.898196 (+ 7704us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:32.898206 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:32.901289 (+ 3083us) server_negotiation.cc:407] Connection header received
0926 15:19:32.901382 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:32.901389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:32.901574 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:32.901770 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:32.902902 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:32.904029 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:32.906762 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:32.908213 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:32.908342 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:32.908497 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:32.908764 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 15:19:32.908939 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7583,"thread_start_us":7442,"threads_started":1}
I20260926 15:19:32.909497 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:32.889701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48930 (local address 127.6.209.194:35653)
0926 15:19:32.900686 (+ 10985us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:32.900697 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:32.900735 (+ 38us) server_negotiation.cc:407] Connection header received
0926 15:19:32.900858 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:32.900867 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:32.901108 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:32.901284 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:32.901988 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:32.903482 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:32.906508 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:32.907923 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:32.908049 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:32.908220 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:32.909070 (+ 850us) server_negotiation.cc:299] Negotiation successful
0926 15:19:32.909225 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10816,"thread_start_us":10722,"threads_started":1}
I20260926 15:19:32.909513 24590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:32.888743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51294 (local address 127.6.209.195:35357)
0926 15:19:32.901461 (+ 12718us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:32.901470 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:32.901504 (+ 34us) server_negotiation.cc:407] Connection header received
0926 15:19:32.901582 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:32.901588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:32.901812 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:32.901979 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:32.902704 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:32.904005 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:32.906613 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:32.907539 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:32.907696 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:32.907939 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:32.909086 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0926 15:19:32.909234 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12576,"thread_start_us":12402,"threads_started":1}
I20260926 15:19:32.909719 24592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:32.889571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:44589 (local address 127.0.0.1:43512)
0926 15:19:32.901030 (+ 11459us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:32.901046 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:32.901171 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:32.901763 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:32.901767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:32.901789 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:32.902674 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:32.902685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:32.904196 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:32.904201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:32.906612 (+ 2411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:32.906623 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:32.908413 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:32.908418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:32.908563 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:32.908622 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 15:19:32.908725 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 15:19:32.909118 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11341,"mutex_wait_us":132,"thread_start_us":11197,"threads_started":1}
I20260926 15:19:32.912496 24379 tablet_service.cc:1467] Processing CreateTablet for tablet 41baef9b483a435294a2f619154e7dd4 (DEFAULT_TABLE table=test-table [id=877fec756afc454481a65bc43725b3ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:19:32.912969 24512 tablet_service.cc:1467] Processing CreateTablet for tablet 41baef9b483a435294a2f619154e7dd4 (DEFAULT_TABLE table=test-table [id=877fec756afc454481a65bc43725b3ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:19:32.912969 24245 tablet_service.cc:1467] Processing CreateTablet for tablet 41baef9b483a435294a2f619154e7dd4 (DEFAULT_TABLE table=test-table [id=877fec756afc454481a65bc43725b3ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:19:32.913650 24379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41baef9b483a435294a2f619154e7dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:32.914147 24512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41baef9b483a435294a2f619154e7dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:32.914146 24245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41baef9b483a435294a2f619154e7dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:32.923411 24595 tablet_bootstrap.cc:492] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb: Bootstrap starting.
I20260926 15:19:32.926200 24595 tablet_bootstrap.cc:654] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:32.926802 24597 tablet_bootstrap.cc:492] T 41baef9b483a435294a2f619154e7dd4 P 6d896f0a3e6543588360a9c614cdd446: Bootstrap starting.
I20260926 15:19:32.927670 24596 tablet_bootstrap.cc:492] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b: Bootstrap starting.
I20260926 15:19:32.928859 24595 log.cc:824] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:32.929744 24597 tablet_bootstrap.cc:654] T 41baef9b483a435294a2f619154e7dd4 P 6d896f0a3e6543588360a9c614cdd446: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:32.930373 24596 tablet_bootstrap.cc:654] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:32.930892 24597 log.cc:824] T 41baef9b483a435294a2f619154e7dd4 P 6d896f0a3e6543588360a9c614cdd446: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:32.931329 24596 log.cc:824] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:32.931754 24595 tablet_bootstrap.cc:492] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb: No bootstrap required, opened a new log
I20260926 15:19:32.931990 24595 ts_tablet_manager.cc:1397] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 15:19:32.933804 24597 tablet_bootstrap.cc:492] T 41baef9b483a435294a2f619154e7dd4 P 6d896f0a3e6543588360a9c614cdd446: No bootstrap required, opened a new log
I20260926 15:19:32.934188 24597 ts_tablet_manager.cc:1397] T 41baef9b483a435294a2f619154e7dd4 P 6d896f0a3e6543588360a9c614cdd446: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 15:19:32.941323 24595 raft_consensus.cc:348] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041e36c1c57548b1863f99be2acbba9b" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35357 } } peers { permanent_uuid: "6d896f0a3e6543588360a9c614cdd446" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 44589 } } peers { permanent_uuid: "09917bcb37ff4f358aeeece71e868bcb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 35653 } }
I20260926 15:19:32.941740 24595 raft_consensus.cc:374] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:32.941843 24595 raft_consensus.cc:729] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09917bcb37ff4f358aeeece71e868bcb, State: Initialized, Role: FOLLOWER
I20260926 15:19:32.943212 24595 consensus_queue.cc:260] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041e36c1c57548b1863f99be2acbba9b" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35357 } } peers { permanent_uuid: "6d896f0a3e6543588360a9c614cdd446" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 44589 } } peers { permanent_uuid: "09917bcb37ff4f358aeeece71e868bcb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 35653 } }
I20260926 15:19:32.944128 24596 tablet_bootstrap.cc:492] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b: No bootstrap required, opened a new log
I20260926 15:19:32.944370 24596 ts_tablet_manager.cc:1397] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.016s
I20260926 15:19:32.945691 24597 raft_consensus.cc:348] T 41baef9b483a435294a2f619154e7dd4 P 6d896f0a3e6543588360a9c614cdd446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041e36c1c57548b1863f99be2acbba9b" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35357 } } peers { permanent_uuid: "6d896f0a3e6543588360a9c614cdd446" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 44589 } } peers { permanent_uuid: "09917bcb37ff4f358aeeece71e868bcb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 35653 } }
I20260926 15:19:32.946393 24597 raft_consensus.cc:374] T 41baef9b483a435294a2f619154e7dd4 P 6d896f0a3e6543588360a9c614cdd446 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:32.946542 24597 raft_consensus.cc:729] T 41baef9b483a435294a2f619154e7dd4 P 6d896f0a3e6543588360a9c614cdd446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d896f0a3e6543588360a9c614cdd446, State: Initialized, Role: FOLLOWER
I20260926 15:19:32.947316 24597 consensus_queue.cc:260] T 41baef9b483a435294a2f619154e7dd4 P 6d896f0a3e6543588360a9c614cdd446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041e36c1c57548b1863f99be2acbba9b" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35357 } } peers { permanent_uuid: "6d896f0a3e6543588360a9c614cdd446" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 44589 } } peers { permanent_uuid: "09917bcb37ff4f358aeeece71e868bcb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 35653 } }
I20260926 15:19:32.951133 24444 heartbeater.cc:503] Master 127.6.209.254:36125 was elected leader, sending a full tablet report...
I20260926 15:19:32.951434 24595 ts_tablet_manager.cc:1428] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb: Time spent starting tablet: real 0.019s user 0.014s sys 0.003s
I20260926 15:19:32.952483 24310 heartbeater.cc:503] Master 127.6.209.254:36125 was elected leader, sending a full tablet report...
I20260926 15:19:32.953279 24597 ts_tablet_manager.cc:1428] T 41baef9b483a435294a2f619154e7dd4 P 6d896f0a3e6543588360a9c614cdd446: Time spent starting tablet: real 0.019s user 0.012s sys 0.003s
I20260926 15:19:32.953838 24596 raft_consensus.cc:348] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041e36c1c57548b1863f99be2acbba9b" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35357 } } peers { permanent_uuid: "6d896f0a3e6543588360a9c614cdd446" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 44589 } } peers { permanent_uuid: "09917bcb37ff4f358aeeece71e868bcb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 35653 } }
I20260926 15:19:32.954197 24596 raft_consensus.cc:374] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:32.954305 24596 raft_consensus.cc:729] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 041e36c1c57548b1863f99be2acbba9b, State: Initialized, Role: FOLLOWER
I20260926 15:19:32.954900 24596 consensus_queue.cc:260] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041e36c1c57548b1863f99be2acbba9b" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35357 } } peers { permanent_uuid: "6d896f0a3e6543588360a9c614cdd446" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 44589 } } peers { permanent_uuid: "09917bcb37ff4f358aeeece71e868bcb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 35653 } }
I20260926 15:19:32.955894 24577 heartbeater.cc:503] Master 127.6.209.254:36125 was elected leader, sending a full tablet report...
I20260926 15:19:32.956539 24596 ts_tablet_manager.cc:1428] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b: Time spent starting tablet: real 0.012s user 0.005s sys 0.006s
W20260926 15:19:33.083115 24578 tablet.cc:2369] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:19:33.131810 24445 tablet.cc:2369] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:19:33.186354 24311 tablet.cc:2369] T 41baef9b483a435294a2f619154e7dd4 P 6d896f0a3e6543588360a9c614cdd446: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:19:33.346648 24601 raft_consensus.cc:482] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:19:33.346951 24601 raft_consensus.cc:504] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041e36c1c57548b1863f99be2acbba9b" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35357 } } peers { permanent_uuid: "6d896f0a3e6543588360a9c614cdd446" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 44589 } } peers { permanent_uuid: "09917bcb37ff4f358aeeece71e868bcb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 35653 } }
I20260926 15:19:33.348498 24601 leader_election.cc:288] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 041e36c1c57548b1863f99be2acbba9b (127.6.209.195:35357), 6d896f0a3e6543588360a9c614cdd446 (127.6.209.193:44589)
I20260926 15:19:33.354449 24590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:33.348777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:45465 (local address 127.6.209.195:35357)
0926 15:19:33.348947 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:33.348951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:33.349451 (+ 500us) server_negotiation.cc:407] Connection header received
0926 15:19:33.349958 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:33.349964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:33.350019 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:33.350108 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:33.350630 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:33.351314 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:33.352930 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:33.353626 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:33.353788 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:33.353854 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:33.354152 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 15:19:33.354242 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 15:19:33.354554 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:33.348734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:35357 (local address 127.6.209.194:45465)
0926 15:19:33.349335 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:33.349352 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:33.349490 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:33.350137 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:33.350141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:33.350173 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:33.350478 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:33.350486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:33.351438 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:33.351446 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:33.352747 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:33.352760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:33.353804 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:33.353810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:33.353993 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:33.354032 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 15:19:33.354120 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 15:19:33.354244 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":332,"threads_started":1}
I20260926 15:19:33.354705 24594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:33.349753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:59007 (local address 127.6.209.193:44589)
0926 15:19:33.349939 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:33.349945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:33.350103 (+ 158us) server_negotiation.cc:407] Connection header received
0926 15:19:33.350364 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:33.350368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:33.350434 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:33.350519 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:33.351040 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:33.351732 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:33.353219 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:33.353884 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:33.354034 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:33.354076 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:33.354377 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 15:19:33.354486 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 15:19:33.354925 24603 raft_consensus.cc:482] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:19:33.355211 24605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:33.349483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:44589 (local address 127.6.209.194:59007)
0926 15:19:33.350012 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:33.350025 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:33.350216 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:33.350529 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:33.350533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:33.350553 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:33.350884 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:33.350892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:33.351873 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:33.351877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:33.353092 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:33.353102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:33.354084 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:33.354091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:33.354246 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:33.354271 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 15:19:33.354346 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 15:19:33.354446 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"spinlock_wait_cycles":40320,"thread_start_us":246,"threads_started":1}
I20260926 15:19:33.355158 24603 raft_consensus.cc:504] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041e36c1c57548b1863f99be2acbba9b" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35357 } } peers { permanent_uuid: "6d896f0a3e6543588360a9c614cdd446" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 44589 } } peers { permanent_uuid: "09917bcb37ff4f358aeeece71e868bcb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 35653 } }
I20260926 15:19:33.355414 24532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41baef9b483a435294a2f619154e7dd4" candidate_uuid: "09917bcb37ff4f358aeeece71e868bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "041e36c1c57548b1863f99be2acbba9b" is_pre_election: true
I20260926 15:19:33.355795 24532 raft_consensus.cc:2454] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09917bcb37ff4f358aeeece71e868bcb in term 0.
I20260926 15:19:33.356251 24265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41baef9b483a435294a2f619154e7dd4" candidate_uuid: "09917bcb37ff4f358aeeece71e868bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d896f0a3e6543588360a9c614cdd446" is_pre_election: true
I20260926 15:19:33.356433 24333 leader_election.cc:302] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 041e36c1c57548b1863f99be2acbba9b, 09917bcb37ff4f358aeeece71e868bcb; no voters:
I20260926 15:19:33.356825 24603 leader_election.cc:288] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d896f0a3e6543588360a9c614cdd446 (127.6.209.193:44589), 09917bcb37ff4f358aeeece71e868bcb (127.6.209.194:35653)
I20260926 15:19:33.356793 24601 raft_consensus.cc:2781] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:19:33.356783 24265 raft_consensus.cc:2454] T 41baef9b483a435294a2f619154e7dd4 P 6d896f0a3e6543588360a9c614cdd446 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09917bcb37ff4f358aeeece71e868bcb in term 0.
I20260926 15:19:33.356961 24601 raft_consensus.cc:482] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:19:33.357064 24601 raft_consensus.cc:3037] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:33.359964 24601 raft_consensus.cc:504] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041e36c1c57548b1863f99be2acbba9b" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35357 } } peers { permanent_uuid: "6d896f0a3e6543588360a9c614cdd446" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 44589 } } peers { permanent_uuid: "09917bcb37ff4f358aeeece71e868bcb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 35653 } }
I20260926 15:19:33.360561 24601 leader_election.cc:288] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb [CANDIDATE]: Term 1 election: Requested vote from peers 041e36c1c57548b1863f99be2acbba9b (127.6.209.195:35357), 6d896f0a3e6543588360a9c614cdd446 (127.6.209.193:44589)
I20260926 15:19:33.360976 24532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41baef9b483a435294a2f619154e7dd4" candidate_uuid: "09917bcb37ff4f358aeeece71e868bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "041e36c1c57548b1863f99be2acbba9b"
I20260926 15:19:33.361122 24265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41baef9b483a435294a2f619154e7dd4" candidate_uuid: "09917bcb37ff4f358aeeece71e868bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d896f0a3e6543588360a9c614cdd446"
I20260926 15:19:33.361171 24532 raft_consensus.cc:3037] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:33.361352 24265 raft_consensus.cc:3037] T 41baef9b483a435294a2f619154e7dd4 P 6d896f0a3e6543588360a9c614cdd446 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:33.364485 24532 raft_consensus.cc:2454] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09917bcb37ff4f358aeeece71e868bcb in term 1.
I20260926 15:19:33.364568 24265 raft_consensus.cc:2454] T 41baef9b483a435294a2f619154e7dd4 P 6d896f0a3e6543588360a9c614cdd446 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09917bcb37ff4f358aeeece71e868bcb in term 1.
I20260926 15:19:33.364951 24333 leader_election.cc:302] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09917bcb37ff4f358aeeece71e868bcb, 6d896f0a3e6543588360a9c614cdd446; no voters:
I20260926 15:19:33.365222 24601 raft_consensus.cc:2781] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:19:33.365757 24601 raft_consensus.cc:686] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb [term 1 LEADER]: Becoming Leader. State: Replica: 09917bcb37ff4f358aeeece71e868bcb, State: Running, Role: LEADER
I20260926 15:19:33.366303 24601 consensus_queue.cc:237] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041e36c1c57548b1863f99be2acbba9b" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35357 } } peers { permanent_uuid: "6d896f0a3e6543588360a9c614cdd446" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 44589 } } peers { permanent_uuid: "09917bcb37ff4f358aeeece71e868bcb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 35653 } }
I20260926 15:19:33.367383 24594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:33.357367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:51131 (local address 127.6.209.193:44589)
0926 15:19:33.357547 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:33.357553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:33.358413 (+ 860us) server_negotiation.cc:407] Connection header received
0926 15:19:33.358662 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:33.358668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:33.358744 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:33.358799 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:33.360875 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:33.361999 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:33.365364 (+ 3365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:33.366436 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:33.366605 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:33.366659 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:33.367034 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 15:19:33.367139 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 15:19:33.367468 24607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:33.357050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:44589 (local address 127.6.209.195:51131)
0926 15:19:33.358321 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:33.358338 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:33.358536 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:33.358973 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:33.358976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:33.358995 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:33.359499 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:33.359507 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:33.363035 (+ 3528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:33.363042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:33.365188 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:33.365201 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:33.366797 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:33.366800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:33.366905 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:33.366924 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 15:19:33.367013 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 15:19:33.367125 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1141,"mutex_wait_us":28,"thread_start_us":1265,"threads_started":1}
I20260926 15:19:33.367950 24265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41baef9b483a435294a2f619154e7dd4" candidate_uuid: "041e36c1c57548b1863f99be2acbba9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d896f0a3e6543588360a9c614cdd446" is_pre_election: true
I20260926 15:19:33.368235 24265 raft_consensus.cc:2379] T 41baef9b483a435294a2f619154e7dd4 P 6d896f0a3e6543588360a9c614cdd446 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 041e36c1c57548b1863f99be2acbba9b in current term 1: Already voted for candidate 09917bcb37ff4f358aeeece71e868bcb in this term.
I20260926 15:19:33.369139 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:33.357515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:47057 (local address 127.6.209.194:35653)
0926 15:19:33.357700 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:33.357704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:33.358608 (+ 904us) server_negotiation.cc:407] Connection header received
0926 15:19:33.358777 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:33.358782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:33.358859 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:33.358941 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:33.359636 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:33.360675 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:33.364634 (+ 3959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:33.365999 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:33.366128 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:33.366191 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:33.368867 (+ 2676us) server_negotiation.cc:299] Negotiation successful
0926 15:19:33.368986 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 15:19:33.369613 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:33.357470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:35653 (local address 127.6.209.195:47057)
0926 15:19:33.358522 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:33.358535 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:33.358625 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:33.358969 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:33.358974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:33.359047 (+ 73us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:33.359487 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:33.359498 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:33.362162 (+ 2664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:33.362167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:33.364459 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:33.364479 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:33.368485 (+ 4006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:33.368490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:33.368673 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:33.368701 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 15:19:33.369277 (+ 576us) client_negotiation.cc:240] Negotiation successful
0926 15:19:33.369378 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"mutex_wait_us":41,"spinlock_wait_cycles":18176,"thread_start_us":573,"threads_started":1}
I20260926 15:19:33.370348 24399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41baef9b483a435294a2f619154e7dd4" candidate_uuid: "041e36c1c57548b1863f99be2acbba9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09917bcb37ff4f358aeeece71e868bcb" is_pre_election: true
I20260926 15:19:33.371287 24465 leader_election.cc:302] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 041e36c1c57548b1863f99be2acbba9b; no voters: 09917bcb37ff4f358aeeece71e868bcb, 6d896f0a3e6543588360a9c614cdd446
I20260926 15:19:33.371579 24603 raft_consensus.cc:2726] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 15:19:33.372735 24120 catalog_manager.cc:5495] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb reported cstate change: term changed from 0 to 1, leader changed from <none> to 09917bcb37ff4f358aeeece71e868bcb (127.6.209.194). New cstate: current_term: 1 leader_uuid: "09917bcb37ff4f358aeeece71e868bcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041e36c1c57548b1863f99be2acbba9b" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d896f0a3e6543588360a9c614cdd446" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 44589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09917bcb37ff4f358aeeece71e868bcb" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 35653 } health_report { overall_health: HEALTHY } } }
I20260926 15:19:33.466161 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:33.461325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48932 (local address 127.6.209.194:35653)
0926 15:19:33.461475 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:33.461480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:33.461883 (+ 403us) server_negotiation.cc:407] Connection header received
0926 15:19:33.462029 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:33.462033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:33.462105 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:33.462209 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:19:33.462517 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:33.463085 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:33.464353 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:33.464671 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:33.464753 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:33.465180 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:19:33.465665 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:19:33.465767 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:33.465869 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 15:19:33.465923 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 15:19:33.472786 24265 raft_consensus.cc:1261] T 41baef9b483a435294a2f619154e7dd4 P 6d896f0a3e6543588360a9c614cdd446 [term 1 FOLLOWER]: Refusing update from remote peer 09917bcb37ff4f358aeeece71e868bcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:19:33.472811 24532 raft_consensus.cc:1261] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b [term 1 FOLLOWER]: Refusing update from remote peer 09917bcb37ff4f358aeeece71e868bcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:19:33.473548 24608 consensus_queue.cc:1035] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "041e36c1c57548b1863f99be2acbba9b" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:19:33.473891 24601 consensus_queue.cc:1035] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d896f0a3e6543588360a9c614cdd446" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 44589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:19:33.485081 24614 mvcc.cc:205] Tried to move back new op lower bound from 7333625747336036352 to 7333625746912350208. Current Snapshot: MvccSnapshot[applied={T|T < 7333625747336036352}]
I20260926 15:19:33.486680 24612 mvcc.cc:205] Tried to move back new op lower bound from 7333625747336036352 to 7333625746912350208. Current Snapshot: MvccSnapshot[applied={T|T < 7333625747336036352 or (T in {7333625747336036352})}]
I20260926 15:19:33.495790 24121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:19:33.496022 24121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:19:33.499018 24121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7cc5c7d3f74c4126a85558022db8d8a3: Sending DeleteTablet for 3 replicas of tablet 41baef9b483a435294a2f619154e7dd4
I20260926 15:19:33.499996 24512 tablet_service.cc:1514] Processing DeleteTablet for tablet 41baef9b483a435294a2f619154e7dd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:19:33 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:51294
I20260926 15:19:33.500144 24245 tablet_service.cc:1514] Processing DeleteTablet for tablet 41baef9b483a435294a2f619154e7dd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:19:33 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:43512
I20260926 15:19:33.500907 24622 tablet_replica.cc:330] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b: stopping tablet replica
I20260926 15:19:33.500967 24379 tablet_service.cc:1514] Processing DeleteTablet for tablet 41baef9b483a435294a2f619154e7dd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:19:33 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:48930
I20260926 15:19:33.501363 24622 raft_consensus.cc:2229] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:19:33.501993 24622 raft_consensus.cc:2258] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:19:33.502863 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 24182
I20260926 15:19:33.511366 24624 tablet_replica.cc:330] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb: stopping tablet replica
I20260926 15:19:33.511848 24624 raft_consensus.cc:2229] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:19:33.512562 24624 raft_consensus.cc:2258] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:19:33.513381 24623 tablet_replica.cc:330] T 41baef9b483a435294a2f619154e7dd4 P 6d896f0a3e6543588360a9c614cdd446: stopping tablet replica
I20260926 15:19:33.513690 24623 raft_consensus.cc:2229] T 41baef9b483a435294a2f619154e7dd4 P 6d896f0a3e6543588360a9c614cdd446 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:19:33.514047 24623 raft_consensus.cc:2258] T 41baef9b483a435294a2f619154e7dd4 P 6d896f0a3e6543588360a9c614cdd446 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:19:33.514286 24624 ts_tablet_manager.cc:1905] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:19:33.514983 24622 ts_tablet_manager.cc:1905] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:19:33.515528 24623 ts_tablet_manager.cc:1905] T 41baef9b483a435294a2f619154e7dd4 P 6d896f0a3e6543588360a9c614cdd446: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:19:33.521462 24624 ts_tablet_manager.cc:1918] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:19:33.521690 24624 log.cc:1196] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/41baef9b483a435294a2f619154e7dd4
I20260926 15:19:33.522265 24624 ts_tablet_manager.cc:1939] T 41baef9b483a435294a2f619154e7dd4 P 09917bcb37ff4f358aeeece71e868bcb: Deleting consensus metadata
I20260926 15:19:33.522236 24623 ts_tablet_manager.cc:1918] T 41baef9b483a435294a2f619154e7dd4 P 6d896f0a3e6543588360a9c614cdd446: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:19:33.522409 24623 log.cc:1196] T 41baef9b483a435294a2f619154e7dd4 P 6d896f0a3e6543588360a9c614cdd446: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-0/wal/wals/41baef9b483a435294a2f619154e7dd4
I20260926 15:19:33.522949 24623 ts_tablet_manager.cc:1939] T 41baef9b483a435294a2f619154e7dd4 P 6d896f0a3e6543588360a9c614cdd446: Deleting consensus metadata
I20260926 15:19:33.523597 24622 ts_tablet_manager.cc:1918] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:19:33.523806 24622 log.cc:1196] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/41baef9b483a435294a2f619154e7dd4
I20260926 15:19:33.524282 24622 ts_tablet_manager.cc:1939] T 41baef9b483a435294a2f619154e7dd4 P 041e36c1c57548b1863f99be2acbba9b: Deleting consensus metadata
I20260926 15:19:33.524751 24106 catalog_manager.cc:4841] TS 6d896f0a3e6543588360a9c614cdd446 (127.6.209.193:44589): tablet 41baef9b483a435294a2f619154e7dd4 (table test-table [id=877fec756afc454481a65bc43725b3ed]) successfully deleted
I20260926 15:19:33.525662 24106 catalog_manager.cc:4841] TS 041e36c1c57548b1863f99be2acbba9b (127.6.209.195:35357): tablet 41baef9b483a435294a2f619154e7dd4 (table test-table [id=877fec756afc454481a65bc43725b3ed]) successfully deleted
I20260926 15:19:33.526760 24105 catalog_manager.cc:4841] TS 09917bcb37ff4f358aeeece71e868bcb (127.6.209.194:35653): tablet 41baef9b483a435294a2f619154e7dd4 (table test-table [id=877fec756afc454481a65bc43725b3ed]) successfully deleted
Sep 26 15:19:33 dist-test-slave-j6q7 krb5kdc[24076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.193@KRBTEST.COM
I20260926 15:19:33.536502 24594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:33.505467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43528 (local address 127.6.209.193:44589)
0926 15:19:33.505625 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:33.505631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:33.523626 (+ 17995us) server_negotiation.cc:407] Connection header received
0926 15:19:33.523745 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:33.523758 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:33.523840 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:33.523935 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:33.525508 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:33.526246 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:33.527575 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:33.527917 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:33.528035 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:33.531594 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:33.531605 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:33.531617 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:33.531646 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:33.534232 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:33.534781 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:33.534788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:33.534789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:33.534901 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:33.535223 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:33.535228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:33.535230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:33.535505 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:33.535773 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:33.536020 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 15:19:33.536235 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 15:19:33.537079 24305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:33.634001 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 24182
I20260926 15:19:33.650624 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 24316
Sep 26 15:19:33 dist-test-slave-j6q7 krb5kdc[24076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.194@KRBTEST.COM
I20260926 15:19:33.666345 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:33.651394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48948 (local address 127.6.209.194:35653)
0926 15:19:33.651643 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:33.651649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:33.651670 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:19:33.651761 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:33.651767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:33.651857 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:33.651949 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:33.653086 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:33.654145 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:33.655265 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:33.655888 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:33.656009 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:33.659779 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:33.659792 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:33.659821 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:33.659874 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:33.663629 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:33.664173 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:33.664182 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:33.664186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:33.664352 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:33.664674 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:33.664679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:33.664683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:33.665281 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:33.665620 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:33.665802 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 15:19:33.666010 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 15:19:33.667019 24439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:33.770777 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 24316
I20260926 15:19:33.787981 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 24449
Sep 26 15:19:33 dist-test-slave-j6q7 krb5kdc[24076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.195@KRBTEST.COM
I20260926 15:19:33.801819 24590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:33.788691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51306 (local address 127.6.209.195:35357)
0926 15:19:33.788914 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:33.788920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:33.788941 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:19:33.789147 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:33.789152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:33.789238 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:33.789317 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:33.790467 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:33.791191 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:33.792519 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:33.793071 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:33.793157 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:33.797268 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:33.797279 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:33.797293 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:33.797320 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:33.799537 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:33.800110 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:33.800116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:33.800118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:33.800239 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:33.800517 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:33.800520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:33.800521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:33.800927 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:33.801128 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:33.801322 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 15:19:33.801462 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 15:19:33.802400 24572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:33.882673 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 24449
I20260926 15:19:33.898201 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 24089
I20260926 15:19:33.899160 24151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:33.997720 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 24089
2026-09-26T15:19:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4258 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T15:19:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:19:34Z Disabled control of system clock
I20260926 15:19:34.064041 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:32807
--webserver_interface=127.6.209.254
--webserver_port=0
--builtin_ntp_servers=127.6.209.212:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:32807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 15:19:34.349926 24633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:34.350152 24633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:34.350219 24633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:34.356298 24633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:19:34.356374 24633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:34.356405 24633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:34.356449 24633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:19:34.356478 24633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:19:34.364279 24633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:32807
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:32807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:34.365269 24633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:34.366695 24633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:34.376861 24639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:34.376829 24641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:34.376899 24638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:34.377547 24633 server_base.cc:1034] running on GCE node
I20260926 15:19:34.378153 24633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:34.379311 24633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:34.380522 24633 hybrid_clock.cc:648] HybridClock initialized: now 1790435974380479 us; error 55 us; skew 500 ppm
I20260926 15:19:34.380913 24633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:19:34.382884 24633 webserver.cc:460] Webserver started at http://127.6.209.254:37155/ using document root <none> and password file <none>
I20260926 15:19:34.383476 24633 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:34.383594 24633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:34.383869 24633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:34.386332 24633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "ea38c723390a4f4da5a8bfe21c5e2f4d"
format_stamp: "Formatted at 2026-09-26 15:19:34 on dist-test-slave-j6q7"
server_key: "8491faaf506fd24909db810f772faed7"
server_key_iv: "5b6e2d27771abe038807ebc58fafebe3"
server_key_version: "encryptionkey@0"
I20260926 15:19:34.387054 24633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "ea38c723390a4f4da5a8bfe21c5e2f4d"
format_stamp: "Formatted at 2026-09-26 15:19:34 on dist-test-slave-j6q7"
server_key: "8491faaf506fd24909db810f772faed7"
server_key_iv: "5b6e2d27771abe038807ebc58fafebe3"
server_key_version: "encryptionkey@0"
I20260926 15:19:34.391513 24633 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 15:19:34.394418 24647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:34.395509 24633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 15:19:34.395714 24633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "ea38c723390a4f4da5a8bfe21c5e2f4d"
format_stamp: "Formatted at 2026-09-26 15:19:34 on dist-test-slave-j6q7"
server_key: "8491faaf506fd24909db810f772faed7"
server_key_iv: "5b6e2d27771abe038807ebc58fafebe3"
server_key_version: "encryptionkey@0"
I20260926 15:19:34.395916 24633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:34.407402 24633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:34.408017 24633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:34.408249 24633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:34.424090 24633 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:32807
I20260926 15:19:34.424116 24698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:32807 every 8 connection(s)
I20260926 15:19:34.425207 24633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:19:34.428296 24699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:34.429452 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 24633
I20260926 15:19:34.429620 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:19:34.430194 6983 external_mini_cluster.cc:1428] Setting key aebbd0857a45f86323f1ab255d0584fd
I20260926 15:19:34.440037 24699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea38c723390a4f4da5a8bfe21c5e2f4d: Bootstrap starting.
I20260926 15:19:34.442533 24702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:34.431989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43848 (local address 127.6.209.254:32807)
0926 15:19:34.432972 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:34.432986 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:34.433019 (+ 33us) server_negotiation.cc:407] Connection header received
0926 15:19:34.434965 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:34.434983 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:34.435342 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:34.435690 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:34.436192 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:34.437419 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:34.438644 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:34.439202 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:34.439384 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:34.440499 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:34.440505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:34.440517 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:34.441071 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:34.441146 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:34.441166 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:34.441300 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:34.441480 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 15:19:34.441742 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":340,"threads_started":1}
I20260926 15:19:34.443145 24699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea38c723390a4f4da5a8bfe21c5e2f4d: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:34.444128 24699 log.cc:824] T 00000000000000000000000000000000 P ea38c723390a4f4da5a8bfe21c5e2f4d: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:34.446480 24699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea38c723390a4f4da5a8bfe21c5e2f4d: No bootstrap required, opened a new log
I20260926 15:19:34.453169 24699 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea38c723390a4f4da5a8bfe21c5e2f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea38c723390a4f4da5a8bfe21c5e2f4d" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 32807 } }
I20260926 15:19:34.453482 24699 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea38c723390a4f4da5a8bfe21c5e2f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:34.453586 24699 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea38c723390a4f4da5a8bfe21c5e2f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea38c723390a4f4da5a8bfe21c5e2f4d, State: Initialized, Role: FOLLOWER
I20260926 15:19:34.454034 24699 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea38c723390a4f4da5a8bfe21c5e2f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea38c723390a4f4da5a8bfe21c5e2f4d" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 32807 } }
I20260926 15:19:34.454242 24699 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea38c723390a4f4da5a8bfe21c5e2f4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:19:34.454376 24699 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea38c723390a4f4da5a8bfe21c5e2f4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:19:34.454523 24699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea38c723390a4f4da5a8bfe21c5e2f4d [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:34.456236 24699 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea38c723390a4f4da5a8bfe21c5e2f4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea38c723390a4f4da5a8bfe21c5e2f4d" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 32807 } }
I20260926 15:19:34.456640 24699 leader_election.cc:302] T 00000000000000000000000000000000 P ea38c723390a4f4da5a8bfe21c5e2f4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea38c723390a4f4da5a8bfe21c5e2f4d; no voters:
I20260926 15:19:34.457161 24699 leader_election.cc:288] T 00000000000000000000000000000000 P ea38c723390a4f4da5a8bfe21c5e2f4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:19:34.457254 24704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea38c723390a4f4da5a8bfe21c5e2f4d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:19:34.457867 24704 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea38c723390a4f4da5a8bfe21c5e2f4d [term 1 LEADER]: Becoming Leader. State: Replica: ea38c723390a4f4da5a8bfe21c5e2f4d, State: Running, Role: LEADER
I20260926 15:19:34.458309 24704 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea38c723390a4f4da5a8bfe21c5e2f4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea38c723390a4f4da5a8bfe21c5e2f4d" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 32807 } }
I20260926 15:19:34.458834 24699 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea38c723390a4f4da5a8bfe21c5e2f4d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:19:34.460935 24706 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea38c723390a4f4da5a8bfe21c5e2f4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea38c723390a4f4da5a8bfe21c5e2f4d. Latest consensus state: current_term: 1 leader_uuid: "ea38c723390a4f4da5a8bfe21c5e2f4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea38c723390a4f4da5a8bfe21c5e2f4d" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 32807 } } }
I20260926 15:19:34.461238 24705 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea38c723390a4f4da5a8bfe21c5e2f4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea38c723390a4f4da5a8bfe21c5e2f4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea38c723390a4f4da5a8bfe21c5e2f4d" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 32807 } } }
I20260926 15:19:34.461464 24706 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea38c723390a4f4da5a8bfe21c5e2f4d [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:34.461535 24705 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea38c723390a4f4da5a8bfe21c5e2f4d [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:34.465339 24714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:19:34.469882 24714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:19:34.476961 24714 catalog_manager.cc:1347] Generated new cluster ID: 1341b6bae67f4367a40dea1db074e8ba
I20260926 15:19:34.477042 24714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:19:34.490175 24714 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:19:34.491278 24714 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:19:34.508625 24714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea38c723390a4f4da5a8bfe21c5e2f4d: Generated new TSK 0
I20260926 15:19:34.509408 24714 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:19:34.522647 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:32807
--builtin_ntp_servers=127.6.209.212:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 15:19:34.850085 24723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:34.850304 24723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:34.850376 24723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:34.856268 24723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:34.856336 24723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:34.856416 24723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:19:34.864744 24723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:32807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:34.865732 24723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:34.867210 24723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:34.878494 24728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:34.878530 24729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:34.878545 24731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:34.879314 24723 server_base.cc:1034] running on GCE node
I20260926 15:19:34.879909 24723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:34.880761 24723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:34.881994 24723 hybrid_clock.cc:648] HybridClock initialized: now 1790435974881945 us; error 48 us; skew 500 ppm
I20260926 15:19:34.882257 24723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:19:34.884083 24723 webserver.cc:460] Webserver started at http://127.6.209.193:34415/ using document root <none> and password file <none>
I20260926 15:19:34.884701 24723 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:34.884845 24723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:34.885139 24723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:34.887576 24723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "de5d0642b2e74a69b07b13acd50cd784"
format_stamp: "Formatted at 2026-09-26 15:19:34 on dist-test-slave-j6q7"
server_key: "82abf95fc6d4ad07938b14505b9c6ab5"
server_key_iv: "ebcd0f97168a80c8d222c756169cae8a"
server_key_version: "encryptionkey@0"
I20260926 15:19:34.888223 24723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "de5d0642b2e74a69b07b13acd50cd784"
format_stamp: "Formatted at 2026-09-26 15:19:34 on dist-test-slave-j6q7"
server_key: "82abf95fc6d4ad07938b14505b9c6ab5"
server_key_iv: "ebcd0f97168a80c8d222c756169cae8a"
server_key_version: "encryptionkey@0"
I20260926 15:19:34.892438 24723 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260926 15:19:34.895412 24737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:34.896482 24723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:19:34.896631 24723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "de5d0642b2e74a69b07b13acd50cd784"
format_stamp: "Formatted at 2026-09-26 15:19:34 on dist-test-slave-j6q7"
server_key: "82abf95fc6d4ad07938b14505b9c6ab5"
server_key_iv: "ebcd0f97168a80c8d222c756169cae8a"
server_key_version: "encryptionkey@0"
I20260926 15:19:34.896816 24723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:34.916287 24723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:34.916940 24723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:34.917183 24723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:34.917920 24723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:19:34.919098 24723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:19:34.919200 24723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:34.919333 24723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:19:34.919414 24723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:34.949250 24723 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:43931
I20260926 15:19:34.949285 24849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:43931 every 8 connection(s)
I20260926 15:19:34.950270 24723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:19:34.959553 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 24723
I20260926 15:19:34.959715 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:19:34.960175 6983 external_mini_cluster.cc:1428] Setting key a881d375ecfe872db9a13e7a71b6409f
I20260926 15:19:34.962013 24852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:34.952552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:51473 (local address 127.6.209.254:32807)
0926 15:19:34.952967 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:34.952972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:34.953883 (+ 911us) server_negotiation.cc:407] Connection header received
0926 15:19:34.955480 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:34.955487 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:34.955566 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:34.955670 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:34.956587 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:34.957417 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:34.959011 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:34.959615 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:34.959774 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:34.961175 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:34.961181 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:34.961185 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:34.961205 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:34.961270 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:34.961286 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:34.961377 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:34.961564 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 15:19:34.961643 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":199,"threads_started":1}
I20260926 15:19:34.962816 24853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:34.953045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:32807 (local address 127.6.209.193:51473)
0926 15:19:34.953746 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:34.953780 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:34.955253 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:34.955882 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:34.955889 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:34.955964 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:19:34.956438 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:34.956452 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:34.957580 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:34.957584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:34.958865 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:34.958876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:34.959867 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:34.959875 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:34.960067 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:34.960802 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:19:34.960805 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:19:34.960894 (+ 89us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:19:34.960989 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:19:34.961382 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:19:34.961390 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:19:34.961396 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 15:19:34.961524 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 15:19:34.961837 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":210,"threads_started":1}
I20260926 15:19:34.962818 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:32807
--builtin_ntp_servers=127.6.209.212:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 15:19:34.964133 24850 heartbeater.cc:347] Connected to a master server at 127.6.209.254:32807
I20260926 15:19:34.964439 24850 heartbeater.cc:464] Registering TS with master...
I20260926 15:19:34.965206 24850 heartbeater.cc:511] Master 127.6.209.254:32807 requested a full tablet report, sending...
I20260926 15:19:34.967160 24664 ts_manager.cc:194] Registered new tserver with Master: de5d0642b2e74a69b07b13acd50cd784 (127.6.209.193:43931)
I20260926 15:19:34.969218 24664 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.209.193:51473
W20260926 15:19:35.247296 24854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:35.247506 24854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:35.247575 24854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:35.253773 24854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:35.253849 24854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:35.254024 24854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:19:35.261965 24854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:32807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:35.262977 24854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:35.264470 24854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:35.275733 24860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:35.275754 24859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:35.275779 24862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:35.276531 24854 server_base.cc:1034] running on GCE node
I20260926 15:19:35.277099 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:35.277884 24854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:35.279090 24854 hybrid_clock.cc:648] HybridClock initialized: now 1790435975279056 us; error 47 us; skew 500 ppm
I20260926 15:19:35.279358 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:19:35.281407 24854 webserver.cc:460] Webserver started at http://127.6.209.194:37819/ using document root <none> and password file <none>
I20260926 15:19:35.282029 24854 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:35.282148 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:35.282419 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:35.284720 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "480e96e3b6544dbb96f8c6b40caaf8a1"
format_stamp: "Formatted at 2026-09-26 15:19:35 on dist-test-slave-j6q7"
server_key: "f401477125b15bb7275f0dd041fa6f04"
server_key_iv: "6dd9e24980ab24946df17496ccc74eb7"
server_key_version: "encryptionkey@0"
I20260926 15:19:35.285472 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "480e96e3b6544dbb96f8c6b40caaf8a1"
format_stamp: "Formatted at 2026-09-26 15:19:35 on dist-test-slave-j6q7"
server_key: "f401477125b15bb7275f0dd041fa6f04"
server_key_iv: "6dd9e24980ab24946df17496ccc74eb7"
server_key_version: "encryptionkey@0"
I20260926 15:19:35.289604 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:19:35.292614 24868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:35.293792 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260926 15:19:35.293968 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "480e96e3b6544dbb96f8c6b40caaf8a1"
format_stamp: "Formatted at 2026-09-26 15:19:35 on dist-test-slave-j6q7"
server_key: "f401477125b15bb7275f0dd041fa6f04"
server_key_iv: "6dd9e24980ab24946df17496ccc74eb7"
server_key_version: "encryptionkey@0"
I20260926 15:19:35.294162 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:35.304667 24854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:35.305389 24854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:35.305661 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:35.306452 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:19:35.307654 24854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:19:35.307775 24854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:35.307893 24854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:19:35.307972 24854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:35.337011 24854 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:46441
I20260926 15:19:35.337071 24980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:46441 every 8 connection(s)
I20260926 15:19:35.338063 24854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:19:35.344918 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 24854
I20260926 15:19:35.345052 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:19:35.345515 6983 external_mini_cluster.cc:1428] Setting key de2b6d5b0f9b719d0d7527fa6bd0452e
I20260926 15:19:35.348192 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:32807
--builtin_ntp_servers=127.6.209.212:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 15:19:35.349087 24852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:35.340114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:50005 (local address 127.6.209.254:32807)
0926 15:19:35.340289 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:35.340294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:35.341399 (+ 1105us) server_negotiation.cc:407] Connection header received
0926 15:19:35.342987 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:35.342991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:35.343052 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:35.343147 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:35.344043 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:35.344661 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:35.346533 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:35.346963 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:35.347104 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:35.348392 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:35.348397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:35.348401 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:35.348419 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:35.348454 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:35.348468 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:35.348547 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:35.348779 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 15:19:35.348894 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 15:19:35.350219 24983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:35.340460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:32807 (local address 127.6.209.194:50005)
0926 15:19:35.341283 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:35.341315 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:35.342769 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:35.343327 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:35.343334 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:35.343401 (+ 67us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:19:35.343902 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:35.343914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:35.344791 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:35.344795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:35.346364 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:35.346377 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:35.347174 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:35.347178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:35.347450 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:35.348066 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:19:35.348070 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:19:35.348111 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:19:35.348189 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:19:35.348566 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:19:35.348577 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:19:35.348587 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 15:19:35.348763 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 15:19:35.349128 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":240,"threads_started":1}
I20260926 15:19:35.351390 24981 heartbeater.cc:347] Connected to a master server at 127.6.209.254:32807
I20260926 15:19:35.351655 24981 heartbeater.cc:464] Registering TS with master...
I20260926 15:19:35.352280 24981 heartbeater.cc:511] Master 127.6.209.254:32807 requested a full tablet report, sending...
I20260926 15:19:35.353416 24664 ts_manager.cc:194] Registered new tserver with Master: 480e96e3b6544dbb96f8c6b40caaf8a1 (127.6.209.194:46441)
I20260926 15:19:35.354080 24664 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.209.194:50005
W20260926 15:19:35.644462 24984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:35.644692 24984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:35.644865 24984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:35.651440 24984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:35.651522 24984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:35.651682 24984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:19:35.659197 24984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:32807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:35.660341 24984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:35.662091 24984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:35.673358 24990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:35.673394 24992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:35.673357 24989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:35.674268 24984 server_base.cc:1034] running on GCE node
I20260926 15:19:35.674755 24984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:35.675518 24984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:35.676720 24984 hybrid_clock.cc:648] HybridClock initialized: now 1790435975676691 us; error 41 us; skew 500 ppm
I20260926 15:19:35.677000 24984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:19:35.679023 24984 webserver.cc:460] Webserver started at http://127.6.209.195:38205/ using document root <none> and password file <none>
I20260926 15:19:35.679596 24984 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:35.679708 24984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:35.680011 24984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:35.682649 24984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "44d27ce0dfdc4a19952edd8d3f00a758"
format_stamp: "Formatted at 2026-09-26 15:19:35 on dist-test-slave-j6q7"
server_key: "1ed1a2f46e5b711200c48a5fd5dffe43"
server_key_iv: "34e1a57bc0316023762b2e33b0e95bbe"
server_key_version: "encryptionkey@0"
I20260926 15:19:35.683375 24984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "44d27ce0dfdc4a19952edd8d3f00a758"
format_stamp: "Formatted at 2026-09-26 15:19:35 on dist-test-slave-j6q7"
server_key: "1ed1a2f46e5b711200c48a5fd5dffe43"
server_key_iv: "34e1a57bc0316023762b2e33b0e95bbe"
server_key_version: "encryptionkey@0"
I20260926 15:19:35.687642 24984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 15:19:35.690613 24998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:35.692024 24984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:19:35.692232 24984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "44d27ce0dfdc4a19952edd8d3f00a758"
format_stamp: "Formatted at 2026-09-26 15:19:35 on dist-test-slave-j6q7"
server_key: "1ed1a2f46e5b711200c48a5fd5dffe43"
server_key_iv: "34e1a57bc0316023762b2e33b0e95bbe"
server_key_version: "encryptionkey@0"
I20260926 15:19:35.692445 24984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:35.706276 24984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:35.706907 24984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:35.707166 24984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:35.707967 24984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:19:35.709307 24984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:19:35.709432 24984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:35.709560 24984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:19:35.709645 24984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:35.740773 24984 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:39455
I20260926 15:19:35.740909 25110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:39455 every 8 connection(s)
I20260926 15:19:35.741986 24984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:19:35.749720 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 24984
I20260926 15:19:35.749861 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:19:35.750255 6983 external_mini_cluster.cc:1428] Setting key 34fb88de44715b382aeea075fff5d469
I20260926 15:19:35.753357 24852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:35.744237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:60017 (local address 127.6.209.254:32807)
0926 15:19:35.744444 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:35.744451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:35.745718 (+ 1267us) server_negotiation.cc:407] Connection header received
0926 15:19:35.747261 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:35.747264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:35.747323 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:35.747402 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:35.748295 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:35.749027 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:35.750563 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:35.751139 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:35.751298 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:35.752557 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:35.752562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:35.752566 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:35.752586 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:35.752618 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:35.752628 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:35.752713 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:35.752953 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 15:19:35.753039 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 15:19:35.754091 25113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:35.744722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:32807 (local address 127.6.209.195:60017)
0926 15:19:35.745576 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:35.745614 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:35.747045 (+ 1431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:35.747575 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:35.747581 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:35.747654 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:19:35.748143 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:35.748157 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:35.749162 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:35.749166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:35.750420 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:35.750431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:35.751370 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:35.751377 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:35.751585 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:35.752288 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:19:35.752290 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:19:35.752323 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:19:35.752383 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:19:35.752713 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:19:35.752723 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:19:35.752741 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 15:19:35.752942 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 15:19:35.753255 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":245,"threads_started":1}
I20260926 15:19:35.755211 25111 heartbeater.cc:347] Connected to a master server at 127.6.209.254:32807
I20260926 15:19:35.755472 25111 heartbeater.cc:464] Registering TS with master...
I20260926 15:19:35.756114 25111 heartbeater.cc:511] Master 127.6.209.254:32807 requested a full tablet report, sending...
I20260926 15:19:35.757311 24664 ts_manager.cc:194] Registered new tserver with Master: 44d27ce0dfdc4a19952edd8d3f00a758 (127.6.209.195:39455)
I20260926 15:19:35.757990 24664 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.209.195:60017
I20260926 15:19:35.764642 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:19:35.772976 24852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:35.768185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43862 (local address 127.6.209.254:32807)
0926 15:19:35.768308 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:35.768312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:35.768713 (+ 401us) server_negotiation.cc:407] Connection header received
0926 15:19:35.768872 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:35.768875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:35.768931 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:35.768999 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:35.769407 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:35.769947 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:35.771029 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:35.771392 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:35.771476 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:35.772436 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:35.772439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:35.772442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:35.772455 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:35.772476 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:35.772485 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:35.772543 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:35.772654 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 15:19:35.772707 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 15:19:35.775789 24664 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:19:35.781719 24664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:19:35.801690 25125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:35.792759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:39455 (local address 127.0.0.1:57272)
0926 15:19:35.793292 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:35.793307 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:35.793396 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:35.794383 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:35.794386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:35.794406 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:35.794763 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:35.794777 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:35.796736 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:35.796742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:35.798888 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:35.798903 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:35.800645 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:35.800651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:35.800916 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:35.800969 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 15:19:35.801159 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 15:19:35.801329 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":17,"thread_start_us":302,"threads_started":1}
I20260926 15:19:35.801800 25127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:35.793348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:46441 (local address 127.0.0.1:33464)
0926 15:19:35.794112 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:35.794129 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:35.794223 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:35.795240 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:35.795247 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:35.795271 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:35.795570 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:35.795598 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:35.797448 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:35.797453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:35.799312 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:35.799323 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:35.801097 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:35.801104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:35.801323 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:35.801359 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 15:19:35.801428 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 15:19:35.801546 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":710,"threads_started":1}
I20260926 15:19:35.802629 25128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:35.793912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.6.209.194:46441)
0926 15:19:35.794689 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:35.794696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:35.794719 (+ 23us) server_negotiation.cc:407] Connection header received
0926 15:19:35.794800 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:35.794806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:35.795043 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:35.795235 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:35.795811 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:35.797300 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:35.799530 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:35.800803 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:35.802043 (+ 1240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:35.802255 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:35.802306 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 15:19:35.802398 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":881,"threads_started":1}
I20260926 15:19:35.802913 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:35.791430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:43931 (local address 127.0.0.1:50948)
0926 15:19:35.792357 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:35.792392 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:35.792563 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:35.793518 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:35.793527 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:35.793562 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:35.793866 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:35.793884 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:35.795651 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:35.795656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:35.799497 (+ 3841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:35.799553 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:35.802259 (+ 2706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:35.802266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:35.802415 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:35.802445 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 15:19:35.802522 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 15:19:35.802625 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":1,"thread_start_us":752,"threads_started":1}
I20260926 15:19:35.802958 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:35.793098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57272 (local address 127.6.209.195:39455)
0926 15:19:35.793894 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:35.793901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:35.793933 (+ 32us) server_negotiation.cc:407] Connection header received
0926 15:19:35.794009 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:35.794015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:35.794197 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:35.795019 (+ 822us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:35.795281 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:35.796624 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:35.799027 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:35.800307 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:35.802203 (+ 1896us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:35.802428 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:35.802488 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 15:19:35.802623 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":502,"threads_started":1}
I20260926 15:19:35.803611 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:35.791738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50948 (local address 127.6.209.193:43931)
0926 15:19:35.792284 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:35.792291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:35.792485 (+ 194us) server_negotiation.cc:407] Connection header received
0926 15:19:35.792755 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:35.792760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:35.793054 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:35.793217 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:35.794079 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:35.795515 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:35.799714 (+ 4199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:35.801738 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:35.801871 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:35.802137 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:35.803160 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0926 15:19:35.803315 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":290,"threads_started":1}
I20260926 15:19:35.805920 25046 tablet_service.cc:1467] Processing CreateTablet for tablet 585863202f9d484e8732846a90e2755b (DEFAULT_TABLE table=test-table [id=df25ec2e30a74e2b8d8adb559f5fd983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:19:35.805943 24916 tablet_service.cc:1467] Processing CreateTablet for tablet 585863202f9d484e8732846a90e2755b (DEFAULT_TABLE table=test-table [id=df25ec2e30a74e2b8d8adb559f5fd983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:19:35.806459 24785 tablet_service.cc:1467] Processing CreateTablet for tablet 585863202f9d484e8732846a90e2755b (DEFAULT_TABLE table=test-table [id=df25ec2e30a74e2b8d8adb559f5fd983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:19:35.807072 25046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 585863202f9d484e8732846a90e2755b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:35.807159 24916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 585863202f9d484e8732846a90e2755b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:35.807528 24785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 585863202f9d484e8732846a90e2755b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:35.816395 25129 tablet_bootstrap.cc:492] T 585863202f9d484e8732846a90e2755b P 44d27ce0dfdc4a19952edd8d3f00a758: Bootstrap starting.
I20260926 15:19:35.819940 25130 tablet_bootstrap.cc:492] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784: Bootstrap starting.
I20260926 15:19:35.820799 25129 tablet_bootstrap.cc:654] T 585863202f9d484e8732846a90e2755b P 44d27ce0dfdc4a19952edd8d3f00a758: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:35.821503 25131 tablet_bootstrap.cc:492] T 585863202f9d484e8732846a90e2755b P 480e96e3b6544dbb96f8c6b40caaf8a1: Bootstrap starting.
I20260926 15:19:35.822990 25129 log.cc:824] T 585863202f9d484e8732846a90e2755b P 44d27ce0dfdc4a19952edd8d3f00a758: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:35.823496 25130 tablet_bootstrap.cc:654] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:35.824370 25131 tablet_bootstrap.cc:654] T 585863202f9d484e8732846a90e2755b P 480e96e3b6544dbb96f8c6b40caaf8a1: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:35.824908 25130 log.cc:824] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:35.825464 25131 log.cc:824] T 585863202f9d484e8732846a90e2755b P 480e96e3b6544dbb96f8c6b40caaf8a1: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:35.826750 25129 tablet_bootstrap.cc:492] T 585863202f9d484e8732846a90e2755b P 44d27ce0dfdc4a19952edd8d3f00a758: No bootstrap required, opened a new log
I20260926 15:19:35.826975 25129 ts_tablet_manager.cc:1397] T 585863202f9d484e8732846a90e2755b P 44d27ce0dfdc4a19952edd8d3f00a758: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20260926 15:19:35.827656 25131 tablet_bootstrap.cc:492] T 585863202f9d484e8732846a90e2755b P 480e96e3b6544dbb96f8c6b40caaf8a1: No bootstrap required, opened a new log
I20260926 15:19:35.827886 25131 ts_tablet_manager.cc:1397] T 585863202f9d484e8732846a90e2755b P 480e96e3b6544dbb96f8c6b40caaf8a1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 15:19:35.830807 25130 tablet_bootstrap.cc:492] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784: No bootstrap required, opened a new log
I20260926 15:19:35.831070 25130 ts_tablet_manager.cc:1397] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260926 15:19:35.834055 25129 raft_consensus.cc:348] T 585863202f9d484e8732846a90e2755b P 44d27ce0dfdc4a19952edd8d3f00a758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5d0642b2e74a69b07b13acd50cd784" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43931 } } peers { permanent_uuid: "44d27ce0dfdc4a19952edd8d3f00a758" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39455 } } peers { permanent_uuid: "480e96e3b6544dbb96f8c6b40caaf8a1" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 46441 } }
I20260926 15:19:35.834372 25129 raft_consensus.cc:374] T 585863202f9d484e8732846a90e2755b P 44d27ce0dfdc4a19952edd8d3f00a758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:35.834476 25129 raft_consensus.cc:729] T 585863202f9d484e8732846a90e2755b P 44d27ce0dfdc4a19952edd8d3f00a758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d27ce0dfdc4a19952edd8d3f00a758, State: Initialized, Role: FOLLOWER
I20260926 15:19:35.834975 25129 consensus_queue.cc:260] T 585863202f9d484e8732846a90e2755b P 44d27ce0dfdc4a19952edd8d3f00a758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5d0642b2e74a69b07b13acd50cd784" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43931 } } peers { permanent_uuid: "44d27ce0dfdc4a19952edd8d3f00a758" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39455 } } peers { permanent_uuid: "480e96e3b6544dbb96f8c6b40caaf8a1" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 46441 } }
I20260926 15:19:35.835887 25111 heartbeater.cc:503] Master 127.6.209.254:32807 was elected leader, sending a full tablet report...
I20260926 15:19:35.836930 25131 raft_consensus.cc:348] T 585863202f9d484e8732846a90e2755b P 480e96e3b6544dbb96f8c6b40caaf8a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5d0642b2e74a69b07b13acd50cd784" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43931 } } peers { permanent_uuid: "44d27ce0dfdc4a19952edd8d3f00a758" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39455 } } peers { permanent_uuid: "480e96e3b6544dbb96f8c6b40caaf8a1" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 46441 } }
I20260926 15:19:35.837335 25131 raft_consensus.cc:374] T 585863202f9d484e8732846a90e2755b P 480e96e3b6544dbb96f8c6b40caaf8a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:35.837455 25131 raft_consensus.cc:729] T 585863202f9d484e8732846a90e2755b P 480e96e3b6544dbb96f8c6b40caaf8a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 480e96e3b6544dbb96f8c6b40caaf8a1, State: Initialized, Role: FOLLOWER
I20260926 15:19:35.837543 25129 ts_tablet_manager.cc:1428] T 585863202f9d484e8732846a90e2755b P 44d27ce0dfdc4a19952edd8d3f00a758: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260926 15:19:35.838054 25131 consensus_queue.cc:260] T 585863202f9d484e8732846a90e2755b P 480e96e3b6544dbb96f8c6b40caaf8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5d0642b2e74a69b07b13acd50cd784" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43931 } } peers { permanent_uuid: "44d27ce0dfdc4a19952edd8d3f00a758" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39455 } } peers { permanent_uuid: "480e96e3b6544dbb96f8c6b40caaf8a1" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 46441 } }
I20260926 15:19:35.840303 24981 heartbeater.cc:503] Master 127.6.209.254:32807 was elected leader, sending a full tablet report...
I20260926 15:19:35.840895 25131 ts_tablet_manager.cc:1428] T 585863202f9d484e8732846a90e2755b P 480e96e3b6544dbb96f8c6b40caaf8a1: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260926 15:19:35.840750 25130 raft_consensus.cc:348] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5d0642b2e74a69b07b13acd50cd784" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43931 } } peers { permanent_uuid: "44d27ce0dfdc4a19952edd8d3f00a758" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39455 } } peers { permanent_uuid: "480e96e3b6544dbb96f8c6b40caaf8a1" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 46441 } }
I20260926 15:19:35.841180 25130 raft_consensus.cc:374] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:35.841254 25130 raft_consensus.cc:729] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de5d0642b2e74a69b07b13acd50cd784, State: Initialized, Role: FOLLOWER
I20260926 15:19:35.841696 25130 consensus_queue.cc:260] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5d0642b2e74a69b07b13acd50cd784" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43931 } } peers { permanent_uuid: "44d27ce0dfdc4a19952edd8d3f00a758" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39455 } } peers { permanent_uuid: "480e96e3b6544dbb96f8c6b40caaf8a1" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 46441 } }
I20260926 15:19:35.842638 24850 heartbeater.cc:503] Master 127.6.209.254:32807 was elected leader, sending a full tablet report...
I20260926 15:19:35.843092 25130 ts_tablet_manager.cc:1428] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260926 15:19:35.952389 24851 tablet.cc:2369] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:19:35.993543 25112 tablet.cc:2369] T 585863202f9d484e8732846a90e2755b P 44d27ce0dfdc4a19952edd8d3f00a758: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:19:36.089802 24982 tablet.cc:2369] T 585863202f9d484e8732846a90e2755b P 480e96e3b6544dbb96f8c6b40caaf8a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:19:36.104698 25137 raft_consensus.cc:482] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:19:36.105003 25137 raft_consensus.cc:504] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5d0642b2e74a69b07b13acd50cd784" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43931 } } peers { permanent_uuid: "44d27ce0dfdc4a19952edd8d3f00a758" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39455 } } peers { permanent_uuid: "480e96e3b6544dbb96f8c6b40caaf8a1" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 46441 } }
I20260926 15:19:36.106549 25137 leader_election.cc:288] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d27ce0dfdc4a19952edd8d3f00a758 (127.6.209.195:39455), 480e96e3b6544dbb96f8c6b40caaf8a1 (127.6.209.194:46441)
I20260926 15:19:36.112897 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:36.106803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:41767 (local address 127.6.209.195:39455)
0926 15:19:36.107024 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:36.107032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:36.107403 (+ 371us) server_negotiation.cc:407] Connection header received
0926 15:19:36.107620 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:36.107628 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:36.107799 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:36.107957 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:36.108616 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:36.109646 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:36.111295 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:36.112026 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:36.112229 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:36.112294 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:36.112548 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 15:19:36.112661 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 15:19:36.112982 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:36.106764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:39455 (local address 127.6.209.193:41767)
0926 15:19:36.107296 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:36.107312 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:36.107473 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:36.107993 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:36.107998 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:36.108035 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:36.108468 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:36.108481 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:36.109767 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:36.109770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:36.111113 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:36.111124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:36.112218 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:36.112223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:36.112404 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:36.112425 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 15:19:36.112515 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 15:19:36.112621 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"spinlock_wait_cycles":21760,"thread_start_us":290,"threads_started":1}
I20260926 15:19:36.113399 25128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:36.107570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:48421 (local address 127.6.209.194:46441)
0926 15:19:36.108400 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:36.108406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:36.108424 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:19:36.108489 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:36.108495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:36.108562 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:36.108645 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:36.109185 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:36.110024 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:36.111756 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:36.112520 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:36.112689 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:36.112748 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:36.113107 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 15:19:36.113227 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666}
I20260926 15:19:36.113871 25139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:36.107430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:46441 (local address 127.6.209.193:48421)
0926 15:19:36.108021 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:36.108037 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:36.108140 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:36.108684 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:36.108688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:36.108709 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:36.109043 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:36.109050 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:36.110158 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:36.110163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:36.111577 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:36.111589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:36.112698 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:36.112704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:36.112910 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:36.112929 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 15:19:36.113566 (+ 637us) client_negotiation.cc:240] Negotiation successful
0926 15:19:36.113674 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":273,"threads_started":1}
I20260926 15:19:36.113845 25066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "585863202f9d484e8732846a90e2755b" candidate_uuid: "de5d0642b2e74a69b07b13acd50cd784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d27ce0dfdc4a19952edd8d3f00a758" is_pre_election: true
I20260926 15:19:36.114392 25066 raft_consensus.cc:2454] T 585863202f9d484e8732846a90e2755b P 44d27ce0dfdc4a19952edd8d3f00a758 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de5d0642b2e74a69b07b13acd50cd784 in term 0.
I20260926 15:19:36.114784 24936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "585863202f9d484e8732846a90e2755b" candidate_uuid: "de5d0642b2e74a69b07b13acd50cd784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "480e96e3b6544dbb96f8c6b40caaf8a1" is_pre_election: true
I20260926 15:19:36.115041 24740 leader_election.cc:302] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d27ce0dfdc4a19952edd8d3f00a758, de5d0642b2e74a69b07b13acd50cd784; no voters:
I20260926 15:19:36.115288 24936 raft_consensus.cc:2454] T 585863202f9d484e8732846a90e2755b P 480e96e3b6544dbb96f8c6b40caaf8a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de5d0642b2e74a69b07b13acd50cd784 in term 0.
I20260926 15:19:36.115447 25137 raft_consensus.cc:2781] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:19:36.115595 25137 raft_consensus.cc:482] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:19:36.115715 25137 raft_consensus.cc:3037] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:36.117688 25137 raft_consensus.cc:504] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5d0642b2e74a69b07b13acd50cd784" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43931 } } peers { permanent_uuid: "44d27ce0dfdc4a19952edd8d3f00a758" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39455 } } peers { permanent_uuid: "480e96e3b6544dbb96f8c6b40caaf8a1" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 46441 } }
I20260926 15:19:36.118188 25137 leader_election.cc:288] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784 [CANDIDATE]: Term 1 election: Requested vote from peers 44d27ce0dfdc4a19952edd8d3f00a758 (127.6.209.195:39455), 480e96e3b6544dbb96f8c6b40caaf8a1 (127.6.209.194:46441)
I20260926 15:19:36.118677 25066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "585863202f9d484e8732846a90e2755b" candidate_uuid: "de5d0642b2e74a69b07b13acd50cd784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d27ce0dfdc4a19952edd8d3f00a758"
I20260926 15:19:36.118712 24936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "585863202f9d484e8732846a90e2755b" candidate_uuid: "de5d0642b2e74a69b07b13acd50cd784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "480e96e3b6544dbb96f8c6b40caaf8a1"
I20260926 15:19:36.118854 25066 raft_consensus.cc:3037] T 585863202f9d484e8732846a90e2755b P 44d27ce0dfdc4a19952edd8d3f00a758 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:36.118873 24936 raft_consensus.cc:3037] T 585863202f9d484e8732846a90e2755b P 480e96e3b6544dbb96f8c6b40caaf8a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:36.120936 24936 raft_consensus.cc:2454] T 585863202f9d484e8732846a90e2755b P 480e96e3b6544dbb96f8c6b40caaf8a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de5d0642b2e74a69b07b13acd50cd784 in term 1.
I20260926 15:19:36.120990 25066 raft_consensus.cc:2454] T 585863202f9d484e8732846a90e2755b P 44d27ce0dfdc4a19952edd8d3f00a758 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de5d0642b2e74a69b07b13acd50cd784 in term 1.
I20260926 15:19:36.121301 24740 leader_election.cc:302] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 480e96e3b6544dbb96f8c6b40caaf8a1, de5d0642b2e74a69b07b13acd50cd784; no voters:
I20260926 15:19:36.121562 25137 raft_consensus.cc:2781] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:19:36.121959 25137 raft_consensus.cc:686] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784 [term 1 LEADER]: Becoming Leader. State: Replica: de5d0642b2e74a69b07b13acd50cd784, State: Running, Role: LEADER
I20260926 15:19:36.122475 25137 consensus_queue.cc:237] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5d0642b2e74a69b07b13acd50cd784" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43931 } } peers { permanent_uuid: "44d27ce0dfdc4a19952edd8d3f00a758" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39455 } } peers { permanent_uuid: "480e96e3b6544dbb96f8c6b40caaf8a1" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 46441 } }
I20260926 15:19:36.126447 24664 catalog_manager.cc:5495] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784 reported cstate change: term changed from 0 to 1, leader changed from <none> to de5d0642b2e74a69b07b13acd50cd784 (127.6.209.193). New cstate: current_term: 1 leader_uuid: "de5d0642b2e74a69b07b13acd50cd784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5d0642b2e74a69b07b13acd50cd784" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 43931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d27ce0dfdc4a19952edd8d3f00a758" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "480e96e3b6544dbb96f8c6b40caaf8a1" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 46441 } health_report { overall_health: UNKNOWN } } }
I20260926 15:19:36.170439 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:36.165151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50964 (local address 127.6.209.193:43931)
0926 15:19:36.165353 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:36.165359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:36.165468 (+ 109us) server_negotiation.cc:407] Connection header received
0926 15:19:36.165698 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:36.165702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:36.165785 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:36.165954 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:19:36.166327 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:36.167052 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:36.168411 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:36.168793 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:36.168933 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:36.169447 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:19:36.169893 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:19:36.170041 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:36.170136 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 15:19:36.170205 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 15:19:36.177495 24936 raft_consensus.cc:1261] T 585863202f9d484e8732846a90e2755b P 480e96e3b6544dbb96f8c6b40caaf8a1 [term 1 FOLLOWER]: Refusing update from remote peer de5d0642b2e74a69b07b13acd50cd784: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:19:36.177491 25066 raft_consensus.cc:1261] T 585863202f9d484e8732846a90e2755b P 44d27ce0dfdc4a19952edd8d3f00a758 [term 1 FOLLOWER]: Refusing update from remote peer de5d0642b2e74a69b07b13acd50cd784: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:19:36.178308 25140 consensus_queue.cc:1035] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784 [LEADER]: Connected to new peer: Peer: permanent_uuid: "480e96e3b6544dbb96f8c6b40caaf8a1" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 46441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:19:36.178646 25137 consensus_queue.cc:1035] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d27ce0dfdc4a19952edd8d3f00a758" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:19:36.204957 24663 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:19:36.205214 24663 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:19:36.208384 24663 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea38c723390a4f4da5a8bfe21c5e2f4d: Sending DeleteTablet for 3 replicas of tablet 585863202f9d484e8732846a90e2755b
I20260926 15:19:36.209551 24785 tablet_service.cc:1514] Processing DeleteTablet for tablet 585863202f9d484e8732846a90e2755b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:19:36 UTC) from {username='slave'} at 127.0.0.1:50948
I20260926 15:19:36.209602 24916 tablet_service.cc:1514] Processing DeleteTablet for tablet 585863202f9d484e8732846a90e2755b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:19:36 UTC) from {username='slave'} at 127.0.0.1:33464
I20260926 15:19:36.210227 25046 tablet_service.cc:1514] Processing DeleteTablet for tablet 585863202f9d484e8732846a90e2755b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:19:36 UTC) from {username='slave'} at 127.0.0.1:57272
I20260926 15:19:36.211043 25155 tablet_replica.cc:330] T 585863202f9d484e8732846a90e2755b P 44d27ce0dfdc4a19952edd8d3f00a758: stopping tablet replica
I20260926 15:19:36.211475 25155 raft_consensus.cc:2229] T 585863202f9d484e8732846a90e2755b P 44d27ce0dfdc4a19952edd8d3f00a758 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:19:36.211948 25155 raft_consensus.cc:2258] T 585863202f9d484e8732846a90e2755b P 44d27ce0dfdc4a19952edd8d3f00a758 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:19:36.212052 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 24723
I20260926 15:19:36.218227 25155 ts_tablet_manager.cc:1905] T 585863202f9d484e8732846a90e2755b P 44d27ce0dfdc4a19952edd8d3f00a758: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:19:36.224126 25154 tablet_replica.cc:330] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784: stopping tablet replica
I20260926 15:19:36.224633 25154 raft_consensus.cc:2229] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:19:36.225390 25154 raft_consensus.cc:2258] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:19:36.226854 25153 tablet_replica.cc:330] T 585863202f9d484e8732846a90e2755b P 480e96e3b6544dbb96f8c6b40caaf8a1: stopping tablet replica
I20260926 15:19:36.228502 25154 ts_tablet_manager.cc:1905] T 585863202f9d484e8732846a90e2755b P de5d0642b2e74a69b07b13acd50cd784: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:19:36.228564 25155 ts_tablet_manager.cc:1918] T 585863202f9d484e8732846a90e2755b P 44d27ce0dfdc4a19952edd8d3f00a758: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:19:36.228736 25155 log.cc:1196] T 585863202f9d484e8732846a90e2755b P 44d27ce0dfdc4a19952edd8d3f00a758: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/585863202f9d484e8732846a90e2755b
I20260926 15:19:36.229279 25155 ts_tablet_manager.cc:1939] T 585863202f9d484e8732846a90e2755b P 44d27ce0dfdc4a19952edd8d3f00a758: Deleting consensus metadata
I20260926 15:19:36.231010 24650 catalog_manager.cc:4841] TS 44d27ce0dfdc4a19952edd8d3f00a758 (127.6.209.195:39455): tablet 585863202f9d484e8732846a90e2755b (table test-table [id=df25ec2e30a74e2b8d8adb559f5fd983]) successfully deleted
I20260926 15:19:36.231604 25153 raft_consensus.cc:2229] T 585863202f9d484e8732846a90e2755b P 480e96e3b6544dbb96f8c6b40caaf8a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:19:36.232072 25153 raft_consensus.cc:2258] T 585863202f9d484e8732846a90e2755b P 480e96e3b6544dbb96f8c6b40caaf8a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:19:36.233783 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:36.213202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50980 (local address 127.6.209.193:43931)
0926 15:19:36.213359 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:36.213365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:36.226037 (+ 12672us) server_negotiation.cc:407] Connection header received
0926 15:19:36.226238 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:36.226243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:36.226354 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:36.226456 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:36.227011 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:36.228030 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:36.229896 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:36.230554 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:36.230662 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:36.232151 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:36.232162 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:36.232177 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:36.232968 (+ 791us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:36.233027 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:36.233049 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:36.233149 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:36.233326 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 15:19:36.233414 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 15:19:36.234575 24845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:36.236372 25153 ts_tablet_manager.cc:1905] T 585863202f9d484e8732846a90e2755b P 480e96e3b6544dbb96f8c6b40caaf8a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:19:36.241508 25153 ts_tablet_manager.cc:1918] T 585863202f9d484e8732846a90e2755b P 480e96e3b6544dbb96f8c6b40caaf8a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:19:36.241616 25153 log.cc:1196] T 585863202f9d484e8732846a90e2755b P 480e96e3b6544dbb96f8c6b40caaf8a1: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/585863202f9d484e8732846a90e2755b
I20260926 15:19:36.242082 25153 ts_tablet_manager.cc:1939] T 585863202f9d484e8732846a90e2755b P 480e96e3b6544dbb96f8c6b40caaf8a1: Deleting consensus metadata
I20260926 15:19:36.243361 24650 catalog_manager.cc:4841] TS 480e96e3b6544dbb96f8c6b40caaf8a1 (127.6.209.194:46441): tablet 585863202f9d484e8732846a90e2755b (table test-table [id=df25ec2e30a74e2b8d8adb559f5fd983]) successfully deleted
I20260926 15:19:36.323676 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 24723
W20260926 15:19:36.340979 24650 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.209.193:43931 (error 108)
I20260926 15:19:36.341293 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 24854
I20260926 15:19:36.343122 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:36.342435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:43931 (local address 127.0.0.1:50996)
0926 15:19:36.342846 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:36.342960 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:43931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":190,"threads_started":1}
W20260926 15:19:36.343369 24650 catalog_manager.cc:4584] TS de5d0642b2e74a69b07b13acd50cd784 (127.6.209.193:43931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 585863202f9d484e8732846a90e2755b: Network error: Client connection negotiation failed: client connection to 127.6.209.193:43931: connect: Connection refused (error 111)
I20260926 15:19:36.348330 25128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:36.342159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33480 (local address 127.6.209.194:46441)
0926 15:19:36.342335 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:36.342341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:36.342452 (+ 111us) server_negotiation.cc:407] Connection header received
0926 15:19:36.342699 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:36.342704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:36.342811 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:36.342896 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:36.343449 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:36.344414 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:36.345550 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:36.346122 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:36.346231 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:36.347081 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:36.347090 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:36.347103 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:36.347716 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:36.347765 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:36.347784 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:36.347879 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:36.347952 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 15:19:36.348033 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 15:19:36.349040 24976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:36.396904 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:36.396462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:43931 (local address 127.0.0.1:51000)
0926 15:19:36.396621 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:36.396717 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:43931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 15:19:36.397789 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:36.397473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:43931 (local address 127.0.0.1:51010)
0926 15:19:36.397605 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:36.397687 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:43931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 15:19:36.429868 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 24854
I20260926 15:19:36.446964 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 24984
I20260926 15:19:36.453235 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:36.447614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57282 (local address 127.6.209.195:39455)
0926 15:19:36.447795 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:36.447801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:36.447835 (+ 34us) server_negotiation.cc:407] Connection header received
0926 15:19:36.448053 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:36.448062 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:36.448152 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:36.448249 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:36.448619 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:36.449324 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:36.450441 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:36.450783 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:36.450908 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:36.451908 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:36.451917 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:36.451932 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:36.452491 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:36.452563 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:36.452580 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:36.452698 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:36.452802 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 15:19:36.452909 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 15:19:36.453819 25106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:36.458881 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:36.458379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:43931 (local address 127.0.0.1:51022)
0926 15:19:36.458610 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:36.458733 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:43931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260926 15:19:36.459798 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:36.459450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:43931 (local address 127.0.0.1:51032)
0926 15:19:36.459600 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:36.459675 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:43931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 15:19:36.533746 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 24984
I20260926 15:19:36.540412 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:36.539953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:43931 (local address 127.0.0.1:51034)
0926 15:19:36.540160 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:36.540288 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:43931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 15:19:36.541630 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:36.541236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:43931 (local address 127.0.0.1:51050)
0926 15:19:36.541415 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:36.541493 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:43931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 15:19:36.549052 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 24633
I20260926 15:19:36.549961 24694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:36.649220 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 24633
2026-09-26T15:19:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2650 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:19:36 dist-test-slave-j6q7 krb5kdc[25165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:19:36 dist-test-slave-j6q7 krb5kdc[25165](info): set up 2 sockets
Sep 26 15:19:36 dist-test-slave-j6q7 krb5kdc[25165](info): commencing operation
krb5kdc: starting...
W20260926 15:19:37.660655 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:19:37 dist-test-slave-j6q7 krb5kdc[25165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:19:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:19:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:19:37.853372 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:43697
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:43697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:19:38.139701 25178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:38.140018 25178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:38.140189 25178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:38.146440 25178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:19:38.146554 25178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:38.146643 25178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:38.146714 25178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:19:38.146770 25178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:19:38.155054 25178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:43697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:43697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:38.156091 25178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:38.157191 25178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:38.167431 25183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:38.167474 25186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:38.167506 25184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:38.168051 25178 server_base.cc:1034] running on GCE node
I20260926 15:19:38.168697 25178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:38.169873 25178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:38.171057 25178 hybrid_clock.cc:648] HybridClock initialized: now 1790435978171024 us; error 37 us; skew 500 ppm
I20260926 15:19:38.171403 25178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:19:38 dist-test-slave-j6q7 krb5kdc[25165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435978, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:19:38.184558 25178 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:19:38.185885 25178 webserver.cc:460] Webserver started at http://127.6.209.254:35407/ using document root <none> and password file <none>
I20260926 15:19:38.186458 25178 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:38.186570 25178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:38.186848 25178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:38.189440 25178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "d2415974198c4a1c8d0ee759a4e2b611"
format_stamp: "Formatted at 2026-09-26 15:19:38 on dist-test-slave-j6q7"
server_key: "6dc636974a0994095fa65ee0c0d39592"
server_key_iv: "7f5875fb222fd2f19c3fe2cf24e08240"
server_key_version: "encryptionkey@0"
I20260926 15:19:38.190075 25178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "d2415974198c4a1c8d0ee759a4e2b611"
format_stamp: "Formatted at 2026-09-26 15:19:38 on dist-test-slave-j6q7"
server_key: "6dc636974a0994095fa65ee0c0d39592"
server_key_iv: "7f5875fb222fd2f19c3fe2cf24e08240"
server_key_version: "encryptionkey@0"
I20260926 15:19:38.194489 25178 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 15:19:38.197383 25193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:38.198437 25178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:19:38.198587 25178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "d2415974198c4a1c8d0ee759a4e2b611"
format_stamp: "Formatted at 2026-09-26 15:19:38 on dist-test-slave-j6q7"
server_key: "6dc636974a0994095fa65ee0c0d39592"
server_key_iv: "7f5875fb222fd2f19c3fe2cf24e08240"
server_key_version: "encryptionkey@0"
I20260926 15:19:38.198753 25178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:38.224674 25178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:38.228585 25178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:38.228906 25178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:38.246629 25178 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:43697
I20260926 15:19:38.246631 25244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:43697 every 8 connection(s)
I20260926 15:19:38.247840 25178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:19:38.250707 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 25178
I20260926 15:19:38.250878 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:19:38.251350 25245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:38.251451 6983 external_mini_cluster.cc:1428] Setting key 47ec1cbd6023be23758c74caeaf9bfb8
I20260926 15:19:38.264148 25245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2415974198c4a1c8d0ee759a4e2b611: Bootstrap starting.
Sep 26 15:19:38 dist-test-slave-j6q7 krb5kdc[25165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:19:38.266781 25245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2415974198c4a1c8d0ee759a4e2b611: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:38.267686 25245 log.cc:824] T 00000000000000000000000000000000 P d2415974198c4a1c8d0ee759a4e2b611: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:38.270008 25245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2415974198c4a1c8d0ee759a4e2b611: No bootstrap required, opened a new log
I20260926 15:19:38.272261 25248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:38.253210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39974 (local address 127.6.209.254:43697)
0926 15:19:38.253943 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:38.253952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:38.253994 (+ 42us) server_negotiation.cc:407] Connection header received
0926 15:19:38.255534 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:38.255551 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:38.255908 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:38.256211 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:38.257800 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:38.258915 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:38.260553 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:38.261301 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:38.261444 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:38.265962 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:38.265969 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:38.265979 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:38.266023 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:38.268748 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:38.269327 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:38.269333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:38.269338 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:38.269471 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:38.269834 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:38.269838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:38.269840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:38.270287 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:38.270595 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:38.271005 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 15:19:38.271391 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":324,"threads_started":1}
I20260926 15:19:38.277012 25245 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2415974198c4a1c8d0ee759a4e2b611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2415974198c4a1c8d0ee759a4e2b611" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 43697 } }
I20260926 15:19:38.277333 25245 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2415974198c4a1c8d0ee759a4e2b611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:38.277426 25245 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2415974198c4a1c8d0ee759a4e2b611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2415974198c4a1c8d0ee759a4e2b611, State: Initialized, Role: FOLLOWER
I20260926 15:19:38.277858 25245 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2415974198c4a1c8d0ee759a4e2b611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2415974198c4a1c8d0ee759a4e2b611" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 43697 } }
I20260926 15:19:38.278060 25245 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2415974198c4a1c8d0ee759a4e2b611 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:19:38.278182 25245 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2415974198c4a1c8d0ee759a4e2b611 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:19:38.278328 25245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2415974198c4a1c8d0ee759a4e2b611 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:38.280205 25245 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2415974198c4a1c8d0ee759a4e2b611 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2415974198c4a1c8d0ee759a4e2b611" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 43697 } }
I20260926 15:19:38.280694 25245 leader_election.cc:302] T 00000000000000000000000000000000 P d2415974198c4a1c8d0ee759a4e2b611 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2415974198c4a1c8d0ee759a4e2b611; no voters:
I20260926 15:19:38.281286 25245 leader_election.cc:288] T 00000000000000000000000000000000 P d2415974198c4a1c8d0ee759a4e2b611 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:19:38.281399 25250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2415974198c4a1c8d0ee759a4e2b611 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:19:38.282042 25250 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2415974198c4a1c8d0ee759a4e2b611 [term 1 LEADER]: Becoming Leader. State: Replica: d2415974198c4a1c8d0ee759a4e2b611, State: Running, Role: LEADER
I20260926 15:19:38.282526 25250 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2415974198c4a1c8d0ee759a4e2b611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2415974198c4a1c8d0ee759a4e2b611" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 43697 } }
I20260926 15:19:38.283058 25245 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2415974198c4a1c8d0ee759a4e2b611 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:19:38.285138 25252 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2415974198c4a1c8d0ee759a4e2b611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2415974198c4a1c8d0ee759a4e2b611. Latest consensus state: current_term: 1 leader_uuid: "d2415974198c4a1c8d0ee759a4e2b611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2415974198c4a1c8d0ee759a4e2b611" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 43697 } } }
I20260926 15:19:38.285576 25252 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2415974198c4a1c8d0ee759a4e2b611 [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:38.285266 25251 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2415974198c4a1c8d0ee759a4e2b611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2415974198c4a1c8d0ee759a4e2b611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2415974198c4a1c8d0ee759a4e2b611" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 43697 } } }
I20260926 15:19:38.285779 25251 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2415974198c4a1c8d0ee759a4e2b611 [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:38.287456 25259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:19:38.292335 25259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:19:38.300872 25259 catalog_manager.cc:1347] Generated new cluster ID: d22cce357dd541018035654e52a0f7df
I20260926 15:19:38.300971 25259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:19:38.320947 25259 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:19:38.322148 25259 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:19:38.335623 25259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2415974198c4a1c8d0ee759a4e2b611: Generated new TSK 0
I20260926 15:19:38.336354 25259 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
I20260926 15:19:38.398929 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:43697
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:19:38.695875 25271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:38.696081 25271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:38.696158 25271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:38.702806 25271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:38.702883 25271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:38.702971 25271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:19:38.710902 25271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:43697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:38.711863 25271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:38.712935 25271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:38.724586 25277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:38.724606 25279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:38.724629 25276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:38.725472 25271 server_base.cc:1034] running on GCE node
I20260926 15:19:38.726013 25271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:38.726816 25271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:38.728017 25271 hybrid_clock.cc:648] HybridClock initialized: now 1790435978727966 us; error 62 us; skew 500 ppm
I20260926 15:19:38.728271 25271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:19:38 dist-test-slave-j6q7 krb5kdc[25165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435978, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:19:38.730898 25271 init.cc:377] Logged in from keytab as kudu/127.6.209.193@KRBTEST.COM (short username kudu)
I20260926 15:19:38.732197 25271 webserver.cc:460] Webserver started at http://127.6.209.193:34261/ using document root <none> and password file <none>
I20260926 15:19:38.732787 25271 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:38.732955 25271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:38.733218 25271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:38.735587 25271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "85cf4b77b4b9428da0526fca56bc9cc2"
format_stamp: "Formatted at 2026-09-26 15:19:38 on dist-test-slave-j6q7"
server_key: "4093a74895cb15f9c9fd1061f81cd4da"
server_key_iv: "62d996625da60fb2fb539d26767c30e3"
server_key_version: "encryptionkey@0"
I20260926 15:19:38.736248 25271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "85cf4b77b4b9428da0526fca56bc9cc2"
format_stamp: "Formatted at 2026-09-26 15:19:38 on dist-test-slave-j6q7"
server_key: "4093a74895cb15f9c9fd1061f81cd4da"
server_key_iv: "62d996625da60fb2fb539d26767c30e3"
server_key_version: "encryptionkey@0"
I20260926 15:19:38.740538 25271 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 15:19:38.743638 25286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:38.744743 25271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260926 15:19:38.744930 25271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "85cf4b77b4b9428da0526fca56bc9cc2"
format_stamp: "Formatted at 2026-09-26 15:19:38 on dist-test-slave-j6q7"
server_key: "4093a74895cb15f9c9fd1061f81cd4da"
server_key_iv: "62d996625da60fb2fb539d26767c30e3"
server_key_version: "encryptionkey@0"
I20260926 15:19:38.745214 25271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:38.766302 25271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:38.769629 25271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:38.769853 25271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:38.770567 25271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:19:38.771853 25271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:19:38.771939 25271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:38.772017 25271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:19:38.772063 25271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:38.800951 25271 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:35351
I20260926 15:19:38.801019 25398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:35351 every 8 connection(s)
I20260926 15:19:38.802052 25271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:19:38.808504 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 25271
I20260926 15:19:38.808660 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:19:38.809190 6983 external_mini_cluster.cc:1428] Setting key 6ab98d62bfe13fd3e3d73a4bd236fef0
Sep 26 15:19:38 dist-test-slave-j6q7 krb5kdc[25165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435978, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:19:38.821122 25401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:38.804311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:35867 (local address 127.6.209.254:43697)
0926 15:19:38.804709 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:38.804713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:38.805566 (+ 853us) server_negotiation.cc:407] Connection header received
0926 15:19:38.807292 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:38.807297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:38.807380 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:38.807487 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:38.809087 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:38.809867 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:38.811942 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:38.812338 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:38.812441 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:38.816129 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:38.816134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:38.816137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:38.816164 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:38.818369 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:38.818971 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:38.818977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:38.818981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:38.819115 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:38.819458 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:38.819462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:38.819464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:38.819714 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:38.819882 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:38.820474 (+ 592us) server_negotiation.cc:299] Negotiation successful
0926 15:19:38.820702 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":225,"threads_started":1}
I20260926 15:19:38.821926 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:38.804710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:43697 (local address 127.6.209.193:35867)
0926 15:19:38.805422 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:38.805456 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:38.807013 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:38.807700 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:38.807707 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:38.808128 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:19:38.808896 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:38.808919 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:38.810034 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:38.810038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:38.811796 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:38.811812 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:38.812560 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:38.812568 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:38.812908 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:38.813563 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:19:38.813594 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:19:38.815946 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:19:38.818511 (+ 2565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:19:38.818516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:19:38.818524 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:19:38.818860 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:19:38.819208 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:19:38.819210 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:19:38.819213 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:19:38.819353 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:19:38.819842 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:19:38.819848 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:19:38.820152 (+ 304us) client_negotiation.cc:769] Sending connection context
0926 15:19:38.820383 (+ 231us) client_negotiation.cc:240] Negotiation successful
0926 15:19:38.820870 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":246,"threads_started":1}
I20260926 15:19:38.823210 25399 heartbeater.cc:347] Connected to a master server at 127.6.209.254:43697
I20260926 15:19:38.823469 25399 heartbeater.cc:464] Registering TS with master...
I20260926 15:19:38.824038 25399 heartbeater.cc:511] Master 127.6.209.254:43697 requested a full tablet report, sending...
I20260926 15:19:38.825893 25210 ts_manager.cc:194] Registered new tserver with Master: 85cf4b77b4b9428da0526fca56bc9cc2 (127.6.209.193:35351)
I20260926 15:19:38.827152 25210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.193@KRBTEST.COM'} at 127.6.209.193:35867
WARNING: no policy specified for kudu/127.6.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
I20260926 15:19:38.855640 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:43697
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:19:39.138084 25405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:39.138332 25405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:39.138468 25405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:39.144708 25405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:39.144850 25405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:39.144990 25405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:19:39.153146 25405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:43697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:39.154192 25405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:39.155395 25405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:39.166951 25410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:39.166929 25413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:39.166900 25411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:39.167498 25405 server_base.cc:1034] running on GCE node
I20260926 15:19:39.167975 25405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:39.168936 25405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:39.170167 25405 hybrid_clock.cc:648] HybridClock initialized: now 1790435979170118 us; error 47 us; skew 500 ppm
I20260926 15:19:39.170550 25405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:19:39 dist-test-slave-j6q7 krb5kdc[25165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435979, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:19:39.173543 25405 init.cc:377] Logged in from keytab as kudu/127.6.209.194@KRBTEST.COM (short username kudu)
I20260926 15:19:39.174675 25405 webserver.cc:460] Webserver started at http://127.6.209.194:37089/ using document root <none> and password file <none>
I20260926 15:19:39.175212 25405 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:39.175333 25405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:39.175598 25405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:39.177940 25405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "3c274af5e09a44a6a24ecc0b657544f7"
format_stamp: "Formatted at 2026-09-26 15:19:39 on dist-test-slave-j6q7"
server_key: "91f737f8bea84b1a3c64a8b00cb1b186"
server_key_iv: "9c7b5c707705a0fa54001c863167204b"
server_key_version: "encryptionkey@0"
I20260926 15:19:39.178587 25405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "3c274af5e09a44a6a24ecc0b657544f7"
format_stamp: "Formatted at 2026-09-26 15:19:39 on dist-test-slave-j6q7"
server_key: "91f737f8bea84b1a3c64a8b00cb1b186"
server_key_iv: "9c7b5c707705a0fa54001c863167204b"
server_key_version: "encryptionkey@0"
I20260926 15:19:39.183110 25405 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260926 15:19:39.185945 25420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:39.187084 25405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:19:39.187220 25405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "3c274af5e09a44a6a24ecc0b657544f7"
format_stamp: "Formatted at 2026-09-26 15:19:39 on dist-test-slave-j6q7"
server_key: "91f737f8bea84b1a3c64a8b00cb1b186"
server_key_iv: "9c7b5c707705a0fa54001c863167204b"
server_key_version: "encryptionkey@0"
I20260926 15:19:39.187376 25405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:39.197656 25405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:39.201303 25405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:39.201591 25405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:39.202503 25405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:19:39.203675 25405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:19:39.203742 25405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:39.203895 25405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:19:39.203943 25405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:39.232338 25405 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:33415
I20260926 15:19:39.232414 25532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:33415 every 8 connection(s)
I20260926 15:19:39.233543 25405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:19:39.243080 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 25405
I20260926 15:19:39.243250 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:19:39.243791 6983 external_mini_cluster.cc:1428] Setting key bbdd1dd294826130164e829a269b9bac
Sep 26 15:19:39 dist-test-slave-j6q7 krb5kdc[25165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435979, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:19:39.253988 25401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:39.235795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:37203 (local address 127.6.209.254:43697)
0926 15:19:39.235948 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:39.235952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:39.236949 (+ 997us) server_negotiation.cc:407] Connection header received
0926 15:19:39.238625 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:39.238631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:39.238718 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:39.238840 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:39.240221 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.240995 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.242510 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.242914 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.243057 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:39.246765 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:39.246773 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:39.246781 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:39.246821 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:39.250553 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:39.251367 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:39.251376 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:39.251382 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:39.251551 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:39.251996 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:39.252003 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:39.252009 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:39.252392 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:39.252546 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:39.253325 (+ 779us) server_negotiation.cc:299] Negotiation successful
0926 15:19:39.253579 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 15:19:39.254839 25535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:39.236150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:43697 (local address 127.6.209.194:37203)
0926 15:19:39.236765 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:39.236797 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:39.238397 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:39.239056 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:39.239063 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:39.239534 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:19:39.240068 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:39.240084 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.241140 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.241145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:39.242346 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:39.242357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.243157 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.243167 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:39.243468 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:39.244132 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:19:39.244159 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:19:39.246554 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:19:39.250687 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:19:39.250695 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:19:39.250708 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:19:39.251217 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:19:39.251670 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:19:39.251675 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:19:39.251679 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:19:39.251878 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:19:39.252525 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:19:39.252533 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:19:39.252910 (+ 377us) client_negotiation.cc:769] Sending connection context
0926 15:19:39.253197 (+ 287us) client_negotiation.cc:240] Negotiation successful
0926 15:19:39.253697 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":244,"threads_started":1}
I20260926 15:19:39.256165 25533 heartbeater.cc:347] Connected to a master server at 127.6.209.254:43697
I20260926 15:19:39.256434 25533 heartbeater.cc:464] Registering TS with master...
I20260926 15:19:39.257121 25533 heartbeater.cc:511] Master 127.6.209.254:43697 requested a full tablet report, sending...
I20260926 15:19:39.258518 25210 ts_manager.cc:194] Registered new tserver with Master: 3c274af5e09a44a6a24ecc0b657544f7 (127.6.209.194:33415)
I20260926 15:19:39.259539 25210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.194@KRBTEST.COM'} at 127.6.209.194:37203
WARNING: no policy specified for kudu/127.6.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
I20260926 15:19:39.293059 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:43697
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:19:39.575592 25538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:39.575816 25538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:39.575896 25538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:39.582211 25538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:39.582309 25538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:39.582442 25538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:19:39.590757 25538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:43697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:39.591835 25538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:39.593003 25538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:39.605225 25544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:39.605311 25543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:39.605320 25546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:39.606042 25538 server_base.cc:1034] running on GCE node
I20260926 15:19:39.606581 25538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:39.607336 25538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:39.608538 25538 hybrid_clock.cc:648] HybridClock initialized: now 1790435979608506 us; error 46 us; skew 500 ppm
I20260926 15:19:39.608800 25538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:19:39 dist-test-slave-j6q7 krb5kdc[25165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435979, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:19:39.611625 25538 init.cc:377] Logged in from keytab as kudu/127.6.209.195@KRBTEST.COM (short username kudu)
I20260926 15:19:39.612946 25538 webserver.cc:460] Webserver started at http://127.6.209.195:42837/ using document root <none> and password file <none>
I20260926 15:19:39.613607 25538 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:39.613736 25538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:39.614045 25538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:39.616585 25538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "26cc077dc54e44adb6b092560d4327e2"
format_stamp: "Formatted at 2026-09-26 15:19:39 on dist-test-slave-j6q7"
server_key: "3f495744f99da79c37ed5964efbe2ad3"
server_key_iv: "b5d507603f2555d9353ca90ecfb55b5c"
server_key_version: "encryptionkey@0"
I20260926 15:19:39.617359 25538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "26cc077dc54e44adb6b092560d4327e2"
format_stamp: "Formatted at 2026-09-26 15:19:39 on dist-test-slave-j6q7"
server_key: "3f495744f99da79c37ed5964efbe2ad3"
server_key_iv: "b5d507603f2555d9353ca90ecfb55b5c"
server_key_version: "encryptionkey@0"
I20260926 15:19:39.621785 25538 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 15:19:39.624756 25553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:39.625969 25538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:19:39.626132 25538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "26cc077dc54e44adb6b092560d4327e2"
format_stamp: "Formatted at 2026-09-26 15:19:39 on dist-test-slave-j6q7"
server_key: "3f495744f99da79c37ed5964efbe2ad3"
server_key_iv: "b5d507603f2555d9353ca90ecfb55b5c"
server_key_version: "encryptionkey@0"
I20260926 15:19:39.626304 25538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:39.654053 25538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:39.657522 25538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:39.657770 25538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:39.658643 25538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:19:39.659878 25538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:19:39.659946 25538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:39.660108 25538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:19:39.660166 25538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:39.690941 25538 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:40331
I20260926 15:19:39.690974 25665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:40331 every 8 connection(s)
I20260926 15:19:39.692221 25538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:19:39.701431 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 25538
I20260926 15:19:39.701591 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:19:39.702121 6983 external_mini_cluster.cc:1428] Setting key 15637d6ed3b78db61dc7734ec59400f9
Sep 26 15:19:39 dist-test-slave-j6q7 krb5kdc[25165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435979, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:19:39.710170 25401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:39.694530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:45791 (local address 127.6.209.254:43697)
0926 15:19:39.694691 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:39.694696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:39.695701 (+ 1005us) server_negotiation.cc:407] Connection header received
0926 15:19:39.697254 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:39.697258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:39.697320 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:39.697402 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:39.698739 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.699544 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.701132 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.701816 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.701943 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:39.705438 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:39.705441 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:39.705445 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:39.705472 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:39.707889 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:39.708460 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:39.708463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:39.708465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:39.708565 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:39.708903 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:39.708908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:39.708912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:39.709215 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:39.709313 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:39.709814 (+ 501us) server_negotiation.cc:299] Negotiation successful
0926 15:19:39.709967 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 15:19:39.710961 25668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:39.694987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:43697 (local address 127.6.209.195:45791)
0926 15:19:39.695560 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:39.695596 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:39.697034 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:39.697576 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:39.697583 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:39.698034 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:19:39.698548 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:39.698563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.699678 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.699681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:39.700956 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:39.700968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.702035 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.702044 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:39.702304 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:39.702958 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:19:39.702978 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:19:39.705285 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:19:39.708016 (+ 2731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:19:39.708021 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:19:39.708030 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:19:39.708371 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:19:39.708640 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:19:39.708643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:19:39.708645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:19:39.708783 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:19:39.709310 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:19:39.709315 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:19:39.709548 (+ 233us) client_negotiation.cc:769] Sending connection context
0926 15:19:39.709720 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 15:19:39.710003 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":236,"threads_started":1}
I20260926 15:19:39.712144 25666 heartbeater.cc:347] Connected to a master server at 127.6.209.254:43697
I20260926 15:19:39.712387 25666 heartbeater.cc:464] Registering TS with master...
I20260926 15:19:39.713013 25666 heartbeater.cc:511] Master 127.6.209.254:43697 requested a full tablet report, sending...
I20260926 15:19:39.714154 25210 ts_manager.cc:194] Registered new tserver with Master: 26cc077dc54e44adb6b092560d4327e2 (127.6.209.195:40331)
I20260926 15:19:39.714787 25210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.195@KRBTEST.COM'} at 127.6.209.195:45791
I20260926 15:19:39.716535 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:19:39.731493 25401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:39.720059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39988 (local address 127.6.209.254:43697)
0926 15:19:39.720190 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:39.720194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:39.720709 (+ 515us) server_negotiation.cc:407] Connection header received
0926 15:19:39.720888 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:39.720891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:39.720944 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:39.721030 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:39.722380 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.723007 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.724065 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.724404 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.724483 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:39.726590 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:39.726594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:39.726597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:39.726621 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:39.729087 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:39.729816 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:39.729819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:39.729821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:39.729991 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:39.730363 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:39.730377 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:39.730379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:39.730614 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:39.730785 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:39.731061 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 15:19:39.731191 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 15:19:39.734690 25210 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:19:39.740427 25210 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:19:39.771311 25680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:39.752192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:33415 (local address 127.0.0.1:45508)
0926 15:19:39.753653 (+ 1461us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:39.753674 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:39.753803 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:39.763969 (+ 10166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:39.763975 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:39.764001 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:39.764456 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:39.764470 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.766289 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.766294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:39.768078 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:39.768089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.770331 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.770337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:39.770510 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:39.770553 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 15:19:39.770681 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 15:19:39.770867 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1237,"mutex_wait_us":2,"thread_start_us":1128,"threads_started":1}
I20260926 15:19:39.771808 25679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:39.752099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44406 (local address 127.6.209.195:40331)
0926 15:19:39.764720 (+ 12621us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:39.764729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:39.764776 (+ 47us) server_negotiation.cc:407] Connection header received
0926 15:19:39.764890 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:39.764897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:39.765100 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:39.765945 (+ 845us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:39.766194 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.767562 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.769344 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.770338 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.770454 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:39.770656 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:39.771417 (+ 761us) server_negotiation.cc:299] Negotiation successful
0926 15:19:39.771581 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12463,"thread_start_us":12313,"threads_started":1}
I20260926 15:19:39.771399 25681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:39.752873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45508 (local address 127.6.209.194:33415)
0926 15:19:39.763327 (+ 10454us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:39.763338 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:39.763375 (+ 37us) server_negotiation.cc:407] Connection header received
0926 15:19:39.763483 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:39.763494 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:39.763777 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:39.763962 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:39.764657 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.766128 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.768220 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.770098 (+ 1878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.770287 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:39.770521 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:39.770750 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 15:19:39.770968 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10226,"thread_start_us":11230,"threads_started":1}
I20260926 15:19:39.772195 25678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:39.751897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:40331 (local address 127.0.0.1:44406)
0926 15:19:39.752648 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:39.752699 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:39.752924 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:39.765308 (+ 12384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:39.765312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:39.765338 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:39.765768 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:39.765779 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.767678 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.767681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:39.769179 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:39.769190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.770799 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.770805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:39.770966 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:39.771003 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 15:19:39.771084 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 15:19:39.771202 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":283,"threads_started":1}
I20260926 15:19:39.775457 25468 tablet_service.cc:1467] Processing CreateTablet for tablet d774bf07ba924a9d8de0e8c67ea4913d (DEFAULT_TABLE table=test-table [id=ff6de552cb98489e99e72d42504bb227]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:19:39.776109 25601 tablet_service.cc:1467] Processing CreateTablet for tablet d774bf07ba924a9d8de0e8c67ea4913d (DEFAULT_TABLE table=test-table [id=ff6de552cb98489e99e72d42504bb227]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:19:39.776613 25468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d774bf07ba924a9d8de0e8c67ea4913d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:39.777349 25601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d774bf07ba924a9d8de0e8c67ea4913d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:39.781970 25682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:39.755099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:35351 (local address 127.0.0.1:47664)
0926 15:19:39.765454 (+ 10355us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:39.765470 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:39.765571 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:39.775702 (+ 10131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:39.775706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:39.775729 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:39.775983 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:39.775993 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.777637 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.777641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:39.779462 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:39.779474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.781212 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.781220 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:39.781416 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:39.781448 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 15:19:39.781545 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 15:19:39.781665 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10226,"thread_start_us":10076,"threads_started":1}
I20260926 15:19:39.782049 25683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:39.755351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.6.209.193:35351)
0926 15:19:39.775150 (+ 19799us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:39.775158 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:39.775183 (+ 25us) server_negotiation.cc:407] Connection header received
0926 15:19:39.775292 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:39.775299 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:39.775523 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:39.775696 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:39.776106 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.777490 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.779609 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.781000 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.781153 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:39.781349 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:39.781595 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 15:19:39.781752 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19651,"thread_start_us":20513,"threads_started":1}
I20260926 15:19:39.785331 25334 tablet_service.cc:1467] Processing CreateTablet for tablet d774bf07ba924a9d8de0e8c67ea4913d (DEFAULT_TABLE table=test-table [id=ff6de552cb98489e99e72d42504bb227]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:19:39.786440 25334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d774bf07ba924a9d8de0e8c67ea4913d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:39.788909 25684 tablet_bootstrap.cc:492] T d774bf07ba924a9d8de0e8c67ea4913d P 26cc077dc54e44adb6b092560d4327e2: Bootstrap starting.
I20260926 15:19:39.790792 25685 tablet_bootstrap.cc:492] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7: Bootstrap starting.
I20260926 15:19:39.793570 25684 tablet_bootstrap.cc:654] T d774bf07ba924a9d8de0e8c67ea4913d P 26cc077dc54e44adb6b092560d4327e2: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:39.794148 25685 tablet_bootstrap.cc:654] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:39.794768 25684 log.cc:824] T d774bf07ba924a9d8de0e8c67ea4913d P 26cc077dc54e44adb6b092560d4327e2: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:39.795339 25685 log.cc:824] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:39.798300 25684 tablet_bootstrap.cc:492] T d774bf07ba924a9d8de0e8c67ea4913d P 26cc077dc54e44adb6b092560d4327e2: No bootstrap required, opened a new log
I20260926 15:19:39.798588 25684 ts_tablet_manager.cc:1397] T d774bf07ba924a9d8de0e8c67ea4913d P 26cc077dc54e44adb6b092560d4327e2: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 15:19:39.798745 25688 tablet_bootstrap.cc:492] T d774bf07ba924a9d8de0e8c67ea4913d P 85cf4b77b4b9428da0526fca56bc9cc2: Bootstrap starting.
I20260926 15:19:39.803134 25688 tablet_bootstrap.cc:654] T d774bf07ba924a9d8de0e8c67ea4913d P 85cf4b77b4b9428da0526fca56bc9cc2: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:39.803434 25685 tablet_bootstrap.cc:492] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7: No bootstrap required, opened a new log
I20260926 15:19:39.803730 25685 ts_tablet_manager.cc:1397] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20260926 15:19:39.804271 25688 log.cc:824] T d774bf07ba924a9d8de0e8c67ea4913d P 85cf4b77b4b9428da0526fca56bc9cc2: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:39.809538 25684 raft_consensus.cc:348] T d774bf07ba924a9d8de0e8c67ea4913d P 26cc077dc54e44adb6b092560d4327e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cc077dc54e44adb6b092560d4327e2" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 40331 } } peers { permanent_uuid: "3c274af5e09a44a6a24ecc0b657544f7" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33415 } } peers { permanent_uuid: "85cf4b77b4b9428da0526fca56bc9cc2" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 35351 } }
I20260926 15:19:39.809986 25684 raft_consensus.cc:374] T d774bf07ba924a9d8de0e8c67ea4913d P 26cc077dc54e44adb6b092560d4327e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:39.810112 25684 raft_consensus.cc:729] T d774bf07ba924a9d8de0e8c67ea4913d P 26cc077dc54e44adb6b092560d4327e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26cc077dc54e44adb6b092560d4327e2, State: Initialized, Role: FOLLOWER
I20260926 15:19:39.810774 25684 consensus_queue.cc:260] T d774bf07ba924a9d8de0e8c67ea4913d P 26cc077dc54e44adb6b092560d4327e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cc077dc54e44adb6b092560d4327e2" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 40331 } } peers { permanent_uuid: "3c274af5e09a44a6a24ecc0b657544f7" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33415 } } peers { permanent_uuid: "85cf4b77b4b9428da0526fca56bc9cc2" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 35351 } }
I20260926 15:19:39.813508 25688 tablet_bootstrap.cc:492] T d774bf07ba924a9d8de0e8c67ea4913d P 85cf4b77b4b9428da0526fca56bc9cc2: No bootstrap required, opened a new log
I20260926 15:19:39.813796 25688 ts_tablet_manager.cc:1397] T d774bf07ba924a9d8de0e8c67ea4913d P 85cf4b77b4b9428da0526fca56bc9cc2: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.011s
I20260926 15:19:39.813575 25685 raft_consensus.cc:348] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cc077dc54e44adb6b092560d4327e2" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 40331 } } peers { permanent_uuid: "3c274af5e09a44a6a24ecc0b657544f7" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33415 } } peers { permanent_uuid: "85cf4b77b4b9428da0526fca56bc9cc2" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 35351 } }
I20260926 15:19:39.814149 25685 raft_consensus.cc:374] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:39.814301 25685 raft_consensus.cc:729] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c274af5e09a44a6a24ecc0b657544f7, State: Initialized, Role: FOLLOWER
I20260926 15:19:39.815104 25685 consensus_queue.cc:260] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cc077dc54e44adb6b092560d4327e2" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 40331 } } peers { permanent_uuid: "3c274af5e09a44a6a24ecc0b657544f7" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33415 } } peers { permanent_uuid: "85cf4b77b4b9428da0526fca56bc9cc2" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 35351 } }
I20260926 15:19:39.816224 25533 heartbeater.cc:503] Master 127.6.209.254:43697 was elected leader, sending a full tablet report...
I20260926 15:19:39.817025 25685 ts_tablet_manager.cc:1428] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260926 15:19:39.818424 25666 heartbeater.cc:503] Master 127.6.209.254:43697 was elected leader, sending a full tablet report...
I20260926 15:19:39.819324 25684 ts_tablet_manager.cc:1428] T d774bf07ba924a9d8de0e8c67ea4913d P 26cc077dc54e44adb6b092560d4327e2: Time spent starting tablet: real 0.021s user 0.011s sys 0.007s
I20260926 15:19:39.822618 25688 raft_consensus.cc:348] T d774bf07ba924a9d8de0e8c67ea4913d P 85cf4b77b4b9428da0526fca56bc9cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cc077dc54e44adb6b092560d4327e2" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 40331 } } peers { permanent_uuid: "3c274af5e09a44a6a24ecc0b657544f7" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33415 } } peers { permanent_uuid: "85cf4b77b4b9428da0526fca56bc9cc2" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 35351 } }
I20260926 15:19:39.822978 25688 raft_consensus.cc:374] T d774bf07ba924a9d8de0e8c67ea4913d P 85cf4b77b4b9428da0526fca56bc9cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:39.823082 25688 raft_consensus.cc:729] T d774bf07ba924a9d8de0e8c67ea4913d P 85cf4b77b4b9428da0526fca56bc9cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85cf4b77b4b9428da0526fca56bc9cc2, State: Initialized, Role: FOLLOWER
I20260926 15:19:39.823632 25688 consensus_queue.cc:260] T d774bf07ba924a9d8de0e8c67ea4913d P 85cf4b77b4b9428da0526fca56bc9cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cc077dc54e44adb6b092560d4327e2" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 40331 } } peers { permanent_uuid: "3c274af5e09a44a6a24ecc0b657544f7" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33415 } } peers { permanent_uuid: "85cf4b77b4b9428da0526fca56bc9cc2" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 35351 } }
I20260926 15:19:39.824646 25399 heartbeater.cc:503] Master 127.6.209.254:43697 was elected leader, sending a full tablet report...
I20260926 15:19:39.825249 25688 ts_tablet_manager.cc:1428] T d774bf07ba924a9d8de0e8c67ea4913d P 85cf4b77b4b9428da0526fca56bc9cc2: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260926 15:19:39.834318 25691 raft_consensus.cc:482] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:19:39.834506 25691 raft_consensus.cc:504] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cc077dc54e44adb6b092560d4327e2" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 40331 } } peers { permanent_uuid: "3c274af5e09a44a6a24ecc0b657544f7" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33415 } } peers { permanent_uuid: "85cf4b77b4b9428da0526fca56bc9cc2" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 35351 } }
I20260926 15:19:39.835716 25691 leader_election.cc:288] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26cc077dc54e44adb6b092560d4327e2 (127.6.209.195:40331), 85cf4b77b4b9428da0526fca56bc9cc2 (127.6.209.193:35351)
I20260926 15:19:39.842245 25679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:39.836093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:58529 (local address 127.6.209.195:40331)
0926 15:19:39.836262 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:39.836266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:39.836790 (+ 524us) server_negotiation.cc:407] Connection header received
0926 15:19:39.837124 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:39.837129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:39.837197 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:39.837271 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:39.837841 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.838491 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.840493 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.841367 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.841541 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:39.841613 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:39.841890 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 15:19:39.842006 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 15:19:39.842414 25693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:39.836060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:40331 (local address 127.6.209.194:58529)
0926 15:19:39.836703 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:39.836722 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:39.836928 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:39.837416 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:39.837419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:39.837442 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:39.837714 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:39.837720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.838600 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.838606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:39.840349 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:39.840360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.841568 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.841574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:39.841723 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:39.841747 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 15:19:39.841855 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 15:19:39.841982 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":1,"thread_start_us":332,"threads_started":1}
I20260926 15:19:39.842551 25683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:39.836990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:54719 (local address 127.6.209.193:35351)
0926 15:19:39.837177 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:39.837183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:39.837421 (+ 238us) server_negotiation.cc:407] Connection header received
0926 15:19:39.837609 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:39.837614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:39.837673 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:39.837776 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:39.838222 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.838832 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.840832 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.841642 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.841814 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:39.841887 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:39.842184 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 15:19:39.842301 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 15:19:39.842732 25694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:39.836787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:35351 (local address 127.6.209.194:54719)
0926 15:19:39.837333 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:39.837345 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:39.837485 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:39.837790 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:39.837794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:39.837816 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:39.838101 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:39.838107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.838996 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.839001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:39.840668 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:39.840679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.841877 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.841883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:39.842035 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:39.842056 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 15:19:39.842162 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 15:19:39.842291 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":249,"threads_started":1}
I20260926 15:19:39.843374 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d774bf07ba924a9d8de0e8c67ea4913d" candidate_uuid: "3c274af5e09a44a6a24ecc0b657544f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26cc077dc54e44adb6b092560d4327e2" is_pre_election: true
I20260926 15:19:39.843667 25354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d774bf07ba924a9d8de0e8c67ea4913d" candidate_uuid: "3c274af5e09a44a6a24ecc0b657544f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85cf4b77b4b9428da0526fca56bc9cc2" is_pre_election: true
I20260926 15:19:39.843757 25621 raft_consensus.cc:2454] T d774bf07ba924a9d8de0e8c67ea4913d P 26cc077dc54e44adb6b092560d4327e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c274af5e09a44a6a24ecc0b657544f7 in term 0.
I20260926 15:19:39.844024 25354 raft_consensus.cc:2454] T d774bf07ba924a9d8de0e8c67ea4913d P 85cf4b77b4b9428da0526fca56bc9cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c274af5e09a44a6a24ecc0b657544f7 in term 0.
I20260926 15:19:39.844280 25422 leader_election.cc:302] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26cc077dc54e44adb6b092560d4327e2, 3c274af5e09a44a6a24ecc0b657544f7; no voters:
I20260926 15:19:39.844579 25691 raft_consensus.cc:2781] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:19:39.844699 25691 raft_consensus.cc:482] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:19:39.844808 25691 raft_consensus.cc:3037] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:39.846668 25691 raft_consensus.cc:504] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cc077dc54e44adb6b092560d4327e2" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 40331 } } peers { permanent_uuid: "3c274af5e09a44a6a24ecc0b657544f7" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33415 } } peers { permanent_uuid: "85cf4b77b4b9428da0526fca56bc9cc2" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 35351 } }
I20260926 15:19:39.847143 25691 leader_election.cc:288] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7 [CANDIDATE]: Term 1 election: Requested vote from peers 26cc077dc54e44adb6b092560d4327e2 (127.6.209.195:40331), 85cf4b77b4b9428da0526fca56bc9cc2 (127.6.209.193:35351)
I20260926 15:19:39.847514 25621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d774bf07ba924a9d8de0e8c67ea4913d" candidate_uuid: "3c274af5e09a44a6a24ecc0b657544f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26cc077dc54e44adb6b092560d4327e2"
I20260926 15:19:39.847554 25354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d774bf07ba924a9d8de0e8c67ea4913d" candidate_uuid: "3c274af5e09a44a6a24ecc0b657544f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85cf4b77b4b9428da0526fca56bc9cc2"
I20260926 15:19:39.847661 25621 raft_consensus.cc:3037] T d774bf07ba924a9d8de0e8c67ea4913d P 26cc077dc54e44adb6b092560d4327e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:39.847718 25354 raft_consensus.cc:3037] T d774bf07ba924a9d8de0e8c67ea4913d P 85cf4b77b4b9428da0526fca56bc9cc2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:39.849794 25621 raft_consensus.cc:2454] T d774bf07ba924a9d8de0e8c67ea4913d P 26cc077dc54e44adb6b092560d4327e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c274af5e09a44a6a24ecc0b657544f7 in term 1.
I20260926 15:19:39.849797 25354 raft_consensus.cc:2454] T d774bf07ba924a9d8de0e8c67ea4913d P 85cf4b77b4b9428da0526fca56bc9cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c274af5e09a44a6a24ecc0b657544f7 in term 1.
I20260926 15:19:39.850099 25422 leader_election.cc:302] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26cc077dc54e44adb6b092560d4327e2, 3c274af5e09a44a6a24ecc0b657544f7; no voters:
I20260926 15:19:39.850344 25691 raft_consensus.cc:2781] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:19:39.850692 25691 raft_consensus.cc:686] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7 [term 1 LEADER]: Becoming Leader. State: Replica: 3c274af5e09a44a6a24ecc0b657544f7, State: Running, Role: LEADER
I20260926 15:19:39.851049 25691 consensus_queue.cc:237] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cc077dc54e44adb6b092560d4327e2" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 40331 } } peers { permanent_uuid: "3c274af5e09a44a6a24ecc0b657544f7" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33415 } } peers { permanent_uuid: "85cf4b77b4b9428da0526fca56bc9cc2" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 35351 } }
I20260926 15:19:39.854919 25210 catalog_manager.cc:5495] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c274af5e09a44a6a24ecc0b657544f7 (127.6.209.194). New cstate: current_term: 1 leader_uuid: "3c274af5e09a44a6a24ecc0b657544f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cc077dc54e44adb6b092560d4327e2" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 40331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c274af5e09a44a6a24ecc0b657544f7" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 33415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85cf4b77b4b9428da0526fca56bc9cc2" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 35351 } health_report { overall_health: UNKNOWN } } }
I20260926 15:19:39.870702 25681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:39.865647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45524 (local address 127.6.209.194:33415)
0926 15:19:39.865846 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:39.865851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:39.866022 (+ 171us) server_negotiation.cc:407] Connection header received
0926 15:19:39.866220 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:39.866223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:39.866278 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:39.866418 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:19:39.866781 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.867376 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.868646 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.869149 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.869269 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:39.869794 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:19:39.870191 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:19:39.870353 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:39.870468 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 15:19:39.870506 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 15:19:39.878266 25621 raft_consensus.cc:1261] T d774bf07ba924a9d8de0e8c67ea4913d P 26cc077dc54e44adb6b092560d4327e2 [term 1 FOLLOWER]: Refusing update from remote peer 3c274af5e09a44a6a24ecc0b657544f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:19:39.878371 25354 raft_consensus.cc:1261] T d774bf07ba924a9d8de0e8c67ea4913d P 85cf4b77b4b9428da0526fca56bc9cc2 [term 1 FOLLOWER]: Refusing update from remote peer 3c274af5e09a44a6a24ecc0b657544f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:19:39.879171 25695 consensus_queue.cc:1035] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26cc077dc54e44adb6b092560d4327e2" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 40331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:19:39.879520 25691 consensus_queue.cc:1035] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85cf4b77b4b9428da0526fca56bc9cc2" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:19:39.888082 25698 mvcc.cc:205] Tried to move back new op lower bound from 7333625773573566464 to 7333625773473320960. Current Snapshot: MvccSnapshot[applied={T|T < 7333625773573566464}]
I20260926 15:19:39.897512 25701 mvcc.cc:205] Tried to move back new op lower bound from 7333625773573566464 to 7333625773473320960. Current Snapshot: MvccSnapshot[applied={T|T < 7333625773573566464}]
I20260926 15:19:39.899168 25210 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:19:39.899374 25210 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:19:39.903488 25210 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2415974198c4a1c8d0ee759a4e2b611: Sending DeleteTablet for 3 replicas of tablet d774bf07ba924a9d8de0e8c67ea4913d
I20260926 15:19:39.904351 25601 tablet_service.cc:1514] Processing DeleteTablet for tablet d774bf07ba924a9d8de0e8c67ea4913d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:19:39 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:44406
I20260926 15:19:39.905422 25468 tablet_service.cc:1514] Processing DeleteTablet for tablet d774bf07ba924a9d8de0e8c67ea4913d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:19:39 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:45508
I20260926 15:19:39.906265 25708 tablet_replica.cc:330] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7: stopping tablet replica
I20260926 15:19:39.906689 25708 raft_consensus.cc:2229] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:19:39.906698 25334 tablet_service.cc:1514] Processing DeleteTablet for tablet d774bf07ba924a9d8de0e8c67ea4913d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:19:39 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:47664
I20260926 15:19:39.907361 25708 raft_consensus.cc:2258] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:19:39.914633 25708 ts_tablet_manager.cc:1905] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:19:39.914939 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 25271
I20260926 15:19:39.923439 25708 ts_tablet_manager.cc:1918] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:19:39.923609 25708 log.cc:1196] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/d774bf07ba924a9d8de0e8c67ea4913d
I20260926 15:19:39.924189 25708 ts_tablet_manager.cc:1939] T d774bf07ba924a9d8de0e8c67ea4913d P 3c274af5e09a44a6a24ecc0b657544f7: Deleting consensus metadata
I20260926 15:19:39.928682 25194 catalog_manager.cc:4841] TS 3c274af5e09a44a6a24ecc0b657544f7 (127.6.209.194:33415): tablet d774bf07ba924a9d8de0e8c67ea4913d (table test-table [id=ff6de552cb98489e99e72d42504bb227]) successfully deleted
I20260926 15:19:39.946148 25709 tablet_replica.cc:330] T d774bf07ba924a9d8de0e8c67ea4913d P 85cf4b77b4b9428da0526fca56bc9cc2: stopping tablet replica
I20260926 15:19:39.946678 25709 raft_consensus.cc:2229] T d774bf07ba924a9d8de0e8c67ea4913d P 85cf4b77b4b9428da0526fca56bc9cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:19:39.947173 25709 raft_consensus.cc:2258] T d774bf07ba924a9d8de0e8c67ea4913d P 85cf4b77b4b9428da0526fca56bc9cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:19:39.953373 25707 tablet_replica.cc:330] T d774bf07ba924a9d8de0e8c67ea4913d P 26cc077dc54e44adb6b092560d4327e2: stopping tablet replica
I20260926 15:19:39.955857 25709 ts_tablet_manager.cc:1905] T d774bf07ba924a9d8de0e8c67ea4913d P 85cf4b77b4b9428da0526fca56bc9cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:19:39.955989 25707 raft_consensus.cc:2229] T d774bf07ba924a9d8de0e8c67ea4913d P 26cc077dc54e44adb6b092560d4327e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:19:39.956868 25707 raft_consensus.cc:2258] T d774bf07ba924a9d8de0e8c67ea4913d P 26cc077dc54e44adb6b092560d4327e2 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 26 15:19:39 dist-test-slave-j6q7 krb5kdc[25165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.193@KRBTEST.COM
I20260926 15:19:39.967190 25709 ts_tablet_manager.cc:1918] T d774bf07ba924a9d8de0e8c67ea4913d P 85cf4b77b4b9428da0526fca56bc9cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:19:39.967367 25709 log.cc:1196] T d774bf07ba924a9d8de0e8c67ea4913d P 85cf4b77b4b9428da0526fca56bc9cc2: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-0/wal/wals/d774bf07ba924a9d8de0e8c67ea4913d
I20260926 15:19:39.967912 25709 ts_tablet_manager.cc:1939] T d774bf07ba924a9d8de0e8c67ea4913d P 85cf4b77b4b9428da0526fca56bc9cc2: Deleting consensus metadata
I20260926 15:19:39.969592 25195 catalog_manager.cc:4841] TS 85cf4b77b4b9428da0526fca56bc9cc2 (127.6.209.193:35351): tablet d774bf07ba924a9d8de0e8c67ea4913d (table test-table [id=ff6de552cb98489e99e72d42504bb227]) successfully deleted
I20260926 15:19:39.970310 25683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:39.915956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47680 (local address 127.6.209.193:35351)
0926 15:19:39.916114 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:39.916119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:39.947851 (+ 31732us) server_negotiation.cc:407] Connection header received
0926 15:19:39.948081 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:39.948086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:39.948155 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:39.948216 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:39.949889 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.950877 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.957036 (+ 6159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:39.957574 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:39.957657 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:39.963063 (+ 5406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:39.963077 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:39.963091 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:39.963124 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:39.966694 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:39.967779 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:39.967786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:39.967789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:39.967923 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:39.968328 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:39.968332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:39.968335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:39.968774 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:39.969095 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:39.969358 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 15:19:39.969992 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 15:19:39.970897 25394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:39.971534 25707 ts_tablet_manager.cc:1905] T d774bf07ba924a9d8de0e8c67ea4913d P 26cc077dc54e44adb6b092560d4327e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:19:39.978583 25707 ts_tablet_manager.cc:1918] T d774bf07ba924a9d8de0e8c67ea4913d P 26cc077dc54e44adb6b092560d4327e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:19:39.978773 25707 log.cc:1196] T d774bf07ba924a9d8de0e8c67ea4913d P 26cc077dc54e44adb6b092560d4327e2: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/d774bf07ba924a9d8de0e8c67ea4913d
I20260926 15:19:39.979276 25707 ts_tablet_manager.cc:1939] T d774bf07ba924a9d8de0e8c67ea4913d P 26cc077dc54e44adb6b092560d4327e2: Deleting consensus metadata
I20260926 15:19:39.980757 25195 catalog_manager.cc:4841] TS 26cc077dc54e44adb6b092560d4327e2 (127.6.209.195:40331): tablet d774bf07ba924a9d8de0e8c67ea4913d (table test-table [id=ff6de552cb98489e99e72d42504bb227]) successfully deleted
I20260926 15:19:40.057040 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 25271
I20260926 15:19:40.078493 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 25405
Sep 26 15:19:40 dist-test-slave-j6q7 krb5kdc[25165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.194@KRBTEST.COM
I20260926 15:19:40.094835 25681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:40.079245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45532 (local address 127.6.209.194:33415)
0926 15:19:40.079419 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:40.079425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:40.079445 (+ 20us) server_negotiation.cc:407] Connection header received
0926 15:19:40.079590 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:40.079595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:40.079675 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:40.079788 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:40.081007 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:40.082295 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:40.083665 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:40.084360 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:40.084486 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:40.088271 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:40.088285 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:40.088313 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:40.088353 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:40.092169 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:40.092790 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:40.092801 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:40.092846 (+ 45us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:40.093021 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:40.093358 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:40.093363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:40.093366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:40.093811 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:40.094093 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:40.094239 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 15:19:40.094448 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 15:19:40.095487 25528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:40.206130 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 25405
I20260926 15:19:40.224663 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 25538
Sep 26 15:19:40 dist-test-slave-j6q7 krb5kdc[25165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.195@KRBTEST.COM
I20260926 15:19:40.239930 25679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:40.225376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44408 (local address 127.6.209.195:40331)
0926 15:19:40.225542 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:40.225546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:40.225565 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:19:40.225756 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:40.225759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:40.225822 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:40.225905 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:40.227112 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:40.227844 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:40.229190 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:40.229626 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:40.229719 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:40.234516 (+ 4797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:40.234530 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:40.234545 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:40.234578 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:40.237652 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:40.238356 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:40.238363 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:40.238365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:40.238492 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:40.238788 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:40.238791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:40.238793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:40.239103 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:40.239318 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:40.239512 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 15:19:40.239663 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 15:19:40.240521 25661 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:40.338632 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 25538
I20260926 15:19:40.354605 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 25178
I20260926 15:19:40.355667 25240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:40.464484 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 25178
2026-09-26T15:19:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (3816 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T15:19:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:19:40Z Disabled control of system clock
I20260926 15:19:40.533327 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:40411
--webserver_interface=127.6.209.254
--webserver_port=0
--builtin_ntp_servers=127.6.209.212:43713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:40411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 15:19:40.843561 25718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:40.843825 25718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:40.843919 25718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:40.850692 25718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:19:40.850782 25718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:40.850823 25718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:40.850868 25718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:19:40.850900 25718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:19:40.859278 25718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:43713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:40411
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:40411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:40.860435 25718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:40.862087 25718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:40.873286 25723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:40.873322 25724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:40.873286 25726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:40.874347 25718 server_base.cc:1034] running on GCE node
I20260926 15:19:40.875108 25718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:40.876394 25718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:40.877635 25718 hybrid_clock.cc:648] HybridClock initialized: now 1790435980877599 us; error 46 us; skew 500 ppm
I20260926 15:19:40.877995 25718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:19:40.880131 25718 webserver.cc:460] Webserver started at http://127.6.209.254:46053/ using document root <none> and password file <none>
I20260926 15:19:40.880791 25718 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:40.880982 25718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:40.881306 25718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:40.883972 25718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "4789cf05bafd4e8fb8c705df4a91728a"
format_stamp: "Formatted at 2026-09-26 15:19:40 on dist-test-slave-j6q7"
server_key: "67110d63a8325af3d46a8d0553f01418"
server_key_iv: "82166e2f6fd1fb7c8c260bb38215b2f1"
server_key_version: "encryptionkey@0"
I20260926 15:19:40.884730 25718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "4789cf05bafd4e8fb8c705df4a91728a"
format_stamp: "Formatted at 2026-09-26 15:19:40 on dist-test-slave-j6q7"
server_key: "67110d63a8325af3d46a8d0553f01418"
server_key_iv: "82166e2f6fd1fb7c8c260bb38215b2f1"
server_key_version: "encryptionkey@0"
I20260926 15:19:40.889431 25718 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260926 15:19:40.892581 25732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:40.893805 25718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260926 15:19:40.893972 25718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "4789cf05bafd4e8fb8c705df4a91728a"
format_stamp: "Formatted at 2026-09-26 15:19:40 on dist-test-slave-j6q7"
server_key: "67110d63a8325af3d46a8d0553f01418"
server_key_iv: "82166e2f6fd1fb7c8c260bb38215b2f1"
server_key_version: "encryptionkey@0"
I20260926 15:19:40.894153 25718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:40.906931 25718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:40.907615 25718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:40.907866 25718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:40.927157 25718 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:40411
I20260926 15:19:40.927182 25783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:40411 every 8 connection(s)
I20260926 15:19:40.928392 25718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:19:40.930414 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 25718
I20260926 15:19:40.930577 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:19:40.931079 6983 external_mini_cluster.cc:1428] Setting key 4d3b2749821870d9fe40a72f79da3e32
I20260926 15:19:40.932437 25784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:40.943431 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:40.932958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44554 (local address 127.6.209.254:40411)
0926 15:19:40.933778 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:40.933787 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:40.933823 (+ 36us) server_negotiation.cc:407] Connection header received
0926 15:19:40.935226 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:40.935245 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:40.935652 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:40.935943 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:40.936500 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:40.937740 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:40.939378 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:40.939961 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:40.940079 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:40.941340 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:40.941347 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:40.941357 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:40.941910 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:40.941984 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:40.942007 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:40.942095 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:40.942309 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 15:19:40.942643 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":304,"threads_started":1}
I20260926 15:19:40.944641 25784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4789cf05bafd4e8fb8c705df4a91728a: Bootstrap starting.
I20260926 15:19:40.947511 25784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4789cf05bafd4e8fb8c705df4a91728a: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:40.948405 25784 log.cc:824] T 00000000000000000000000000000000 P 4789cf05bafd4e8fb8c705df4a91728a: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:40.950649 25784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4789cf05bafd4e8fb8c705df4a91728a: No bootstrap required, opened a new log
I20260926 15:19:40.957062 25784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4789cf05bafd4e8fb8c705df4a91728a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4789cf05bafd4e8fb8c705df4a91728a" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40411 } }
I20260926 15:19:40.957362 25784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4789cf05bafd4e8fb8c705df4a91728a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:40.957458 25784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4789cf05bafd4e8fb8c705df4a91728a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4789cf05bafd4e8fb8c705df4a91728a, State: Initialized, Role: FOLLOWER
I20260926 15:19:40.958050 25784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4789cf05bafd4e8fb8c705df4a91728a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4789cf05bafd4e8fb8c705df4a91728a" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40411 } }
I20260926 15:19:40.958288 25784 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4789cf05bafd4e8fb8c705df4a91728a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:19:40.958449 25784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4789cf05bafd4e8fb8c705df4a91728a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:19:40.958616 25784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4789cf05bafd4e8fb8c705df4a91728a [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:40.961045 25784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4789cf05bafd4e8fb8c705df4a91728a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4789cf05bafd4e8fb8c705df4a91728a" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40411 } }
I20260926 15:19:40.961550 25784 leader_election.cc:302] T 00000000000000000000000000000000 P 4789cf05bafd4e8fb8c705df4a91728a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4789cf05bafd4e8fb8c705df4a91728a; no voters:
I20260926 15:19:40.962183 25784 leader_election.cc:288] T 00000000000000000000000000000000 P 4789cf05bafd4e8fb8c705df4a91728a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:19:40.962328 25789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4789cf05bafd4e8fb8c705df4a91728a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:19:40.963040 25789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4789cf05bafd4e8fb8c705df4a91728a [term 1 LEADER]: Becoming Leader. State: Replica: 4789cf05bafd4e8fb8c705df4a91728a, State: Running, Role: LEADER
I20260926 15:19:40.963574 25789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4789cf05bafd4e8fb8c705df4a91728a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4789cf05bafd4e8fb8c705df4a91728a" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40411 } }
I20260926 15:19:40.964231 25784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4789cf05bafd4e8fb8c705df4a91728a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:19:40.966126 25790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4789cf05bafd4e8fb8c705df4a91728a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4789cf05bafd4e8fb8c705df4a91728a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4789cf05bafd4e8fb8c705df4a91728a" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40411 } } }
I20260926 15:19:40.966176 25791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4789cf05bafd4e8fb8c705df4a91728a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4789cf05bafd4e8fb8c705df4a91728a. Latest consensus state: current_term: 1 leader_uuid: "4789cf05bafd4e8fb8c705df4a91728a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4789cf05bafd4e8fb8c705df4a91728a" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40411 } } }
I20260926 15:19:40.966522 25790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4789cf05bafd4e8fb8c705df4a91728a [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:40.966553 25791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4789cf05bafd4e8fb8c705df4a91728a [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:40.968935 25796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:19:40.974764 25796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:19:40.983470 25796 catalog_manager.cc:1347] Generated new cluster ID: 9c32276975684c9c92702400ef3549e1
I20260926 15:19:40.983563 25796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:19:41.007983 25796 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:19:41.009460 25796 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:19:41.014458 25796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4789cf05bafd4e8fb8c705df4a91728a: Generated new TSK 0
I20260926 15:19:41.015112 25796 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:19:41.022197 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:40411
--builtin_ntp_servers=127.6.209.212:43713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 15:19:41.320495 25808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:41.320724 25808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:41.320870 25808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:41.327039 25808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:41.327111 25808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:41.327234 25808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:19:41.335052 25808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:43713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:40411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:41.336035 25808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:41.337457 25808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:41.348551 25813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:41.348585 25816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:41.348639 25814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:41.349442 25808 server_base.cc:1034] running on GCE node
I20260926 15:19:41.349913 25808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:41.350630 25808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:41.351838 25808 hybrid_clock.cc:648] HybridClock initialized: now 1790435981351799 us; error 49 us; skew 500 ppm
I20260926 15:19:41.352138 25808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:19:41.354038 25808 webserver.cc:460] Webserver started at http://127.6.209.193:42973/ using document root <none> and password file <none>
I20260926 15:19:41.354581 25808 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:41.354691 25808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:41.354959 25808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:41.357375 25808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "aadff1e71c214e289335453ad94b7758"
format_stamp: "Formatted at 2026-09-26 15:19:41 on dist-test-slave-j6q7"
server_key: "619e74002d83307e2b0017d952a06d8a"
server_key_iv: "1bb241623cdad61855ab5a7053d15462"
server_key_version: "encryptionkey@0"
I20260926 15:19:41.358048 25808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "aadff1e71c214e289335453ad94b7758"
format_stamp: "Formatted at 2026-09-26 15:19:41 on dist-test-slave-j6q7"
server_key: "619e74002d83307e2b0017d952a06d8a"
server_key_iv: "1bb241623cdad61855ab5a7053d15462"
server_key_version: "encryptionkey@0"
I20260926 15:19:41.362378 25808 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 15:19:41.365339 25822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:41.366523 25808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260926 15:19:41.366667 25808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "aadff1e71c214e289335453ad94b7758"
format_stamp: "Formatted at 2026-09-26 15:19:41 on dist-test-slave-j6q7"
server_key: "619e74002d83307e2b0017d952a06d8a"
server_key_iv: "1bb241623cdad61855ab5a7053d15462"
server_key_version: "encryptionkey@0"
I20260926 15:19:41.366863 25808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:41.386031 25808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:41.386632 25808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:41.386888 25808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:41.387615 25808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:19:41.388754 25808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:19:41.388891 25808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:41.388994 25808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:19:41.389081 25808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:41.416572 25808 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:35013
I20260926 15:19:41.416666 25934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:35013 every 8 connection(s)
I20260926 15:19:41.417585 25808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:19:41.423173 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 25808
I20260926 15:19:41.423347 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:19:41.423799 6983 external_mini_cluster.cc:1428] Setting key 4bb45e2a07a91a54012a3df3788a47a0
I20260926 15:19:41.426311 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:40411
--builtin_ntp_servers=127.6.209.212:43713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 15:19:41.429059 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:41.419746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:55771 (local address 127.6.209.254:40411)
0926 15:19:41.419938 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:41.419942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:41.421110 (+ 1168us) server_negotiation.cc:407] Connection header received
0926 15:19:41.422726 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:41.422730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:41.422801 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:41.422874 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:41.424026 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:41.424828 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:41.426706 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:41.427043 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:41.427125 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:41.428439 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:41.428443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:41.428445 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:41.428460 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:41.428488 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:41.428498 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:41.428558 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:41.428767 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 15:19:41.428869 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 15:19:41.430181 25937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:41.420170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40411 (local address 127.6.209.193:55771)
0926 15:19:41.420970 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:41.421000 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:41.422485 (+ 1485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:41.423112 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:41.423122 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:41.423230 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:19:41.423869 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:41.423892 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:41.424960 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:41.424963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:41.426575 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:41.426590 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:41.427236 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:41.427245 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:41.427525 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:41.428141 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:19:41.428146 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:19:41.428187 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:19:41.428269 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:19:41.428585 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:19:41.428595 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:19:41.428604 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 15:19:41.428760 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 15:19:41.429083 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":237,"threads_started":1}
I20260926 15:19:41.431473 25935 heartbeater.cc:347] Connected to a master server at 127.6.209.254:40411
I20260926 15:19:41.431759 25935 heartbeater.cc:464] Registering TS with master...
I20260926 15:19:41.432394 25935 heartbeater.cc:511] Master 127.6.209.254:40411 requested a full tablet report, sending...
I20260926 15:19:41.434222 25749 ts_manager.cc:194] Registered new tserver with Master: aadff1e71c214e289335453ad94b7758 (127.6.209.193:35013)
I20260926 15:19:41.435945 25749 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.209.193:55771
W20260926 15:19:41.711730 25938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:41.711966 25938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:41.712045 25938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:41.718161 25938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:41.718238 25938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:41.718330 25938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:19:41.726121 25938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:43713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:40411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:41.727074 25938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:41.728502 25938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:41.739578 25944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:41.739559 25943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:41.739553 25946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:41.740191 25938 server_base.cc:1034] running on GCE node
I20260926 15:19:41.740734 25938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:41.741577 25938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:41.742776 25938 hybrid_clock.cc:648] HybridClock initialized: now 1790435981742731 us; error 40 us; skew 500 ppm
I20260926 15:19:41.743070 25938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:19:41.744984 25938 webserver.cc:460] Webserver started at http://127.6.209.194:36419/ using document root <none> and password file <none>
I20260926 15:19:41.745532 25938 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:41.745637 25938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:41.745920 25938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:41.748198 25938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "d09fc6da80214d32bf8b39f2bf5d6c25"
format_stamp: "Formatted at 2026-09-26 15:19:41 on dist-test-slave-j6q7"
server_key: "604ca579a17abb610438259b48badf6b"
server_key_iv: "da019b486954864d0420a3b93645296a"
server_key_version: "encryptionkey@0"
I20260926 15:19:41.748896 25938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "d09fc6da80214d32bf8b39f2bf5d6c25"
format_stamp: "Formatted at 2026-09-26 15:19:41 on dist-test-slave-j6q7"
server_key: "604ca579a17abb610438259b48badf6b"
server_key_iv: "da019b486954864d0420a3b93645296a"
server_key_version: "encryptionkey@0"
I20260926 15:19:41.753100 25938 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 15:19:41.756151 25952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:41.757632 25938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:19:41.757776 25938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "d09fc6da80214d32bf8b39f2bf5d6c25"
format_stamp: "Formatted at 2026-09-26 15:19:41 on dist-test-slave-j6q7"
server_key: "604ca579a17abb610438259b48badf6b"
server_key_iv: "da019b486954864d0420a3b93645296a"
server_key_version: "encryptionkey@0"
I20260926 15:19:41.757968 25938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:41.777472 25938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:41.778072 25938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:41.778288 25938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:41.779044 25938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:19:41.780308 25938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:19:41.780387 25938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:41.780453 25938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:19:41.780504 25938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:41.807381 25938 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:36285
I20260926 15:19:41.807466 26064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:36285 every 8 connection(s)
I20260926 15:19:41.808398 25938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:19:41.809604 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 25938
I20260926 15:19:41.809760 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:19:41.810127 6983 external_mini_cluster.cc:1428] Setting key 4a668f538b50914b2e120fb16290f541
I20260926 15:19:41.813482 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:40411
--builtin_ntp_servers=127.6.209.212:43713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 15:19:41.820092 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:41.810731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:51995 (local address 127.6.209.254:40411)
0926 15:19:41.810888 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:41.810892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:41.812108 (+ 1216us) server_negotiation.cc:407] Connection header received
0926 15:19:41.813950 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:41.813955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:41.814061 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:41.814155 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:41.815180 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:41.816118 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:41.817453 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:41.818105 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:41.818225 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:41.819277 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:41.819284 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:41.819291 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:41.819312 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:41.819352 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:41.819368 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:41.819447 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:41.819612 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 15:19:41.819707 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:19:41.821112 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:41.811126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40411 (local address 127.6.209.194:51995)
0926 15:19:41.811974 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:41.812011 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:41.813702 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:41.814443 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:41.814450 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:41.814531 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:19:41.815029 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:41.815043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:41.816246 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:41.816248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:41.817327 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:41.817338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:41.818273 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:41.818277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:41.818444 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:41.819014 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:19:41.819016 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:19:41.819046 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:19:41.819109 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:19:41.819449 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:19:41.819455 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:19:41.819460 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:19:41.819573 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 15:19:41.819805 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"spinlock_wait_cycles":12032,"thread_start_us":353,"threads_started":1}
I20260926 15:19:41.822299 26065 heartbeater.cc:347] Connected to a master server at 127.6.209.254:40411
I20260926 15:19:41.822575 26065 heartbeater.cc:464] Registering TS with master...
I20260926 15:19:41.823328 26065 heartbeater.cc:511] Master 127.6.209.254:40411 requested a full tablet report, sending...
I20260926 15:19:41.824621 25749 ts_manager.cc:194] Registered new tserver with Master: d09fc6da80214d32bf8b39f2bf5d6c25 (127.6.209.194:36285)
I20260926 15:19:41.825405 25749 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.209.194:51995
W20260926 15:19:42.106348 26068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:42.106601 26068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:42.106755 26068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:42.113095 26068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:42.113205 26068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:42.113360 26068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:19:42.121029 26068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:43713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:40411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:42.122067 26068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:42.123608 26068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:42.134814 26076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:42.134891 26073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:42.134869 26074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:42.135543 26068 server_base.cc:1034] running on GCE node
I20260926 15:19:42.136040 26068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:42.136765 26068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:42.137979 26068 hybrid_clock.cc:648] HybridClock initialized: now 1790435982137933 us; error 42 us; skew 500 ppm
I20260926 15:19:42.138250 26068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:19:42.140257 26068 webserver.cc:460] Webserver started at http://127.6.209.195:33123/ using document root <none> and password file <none>
I20260926 15:19:42.140941 26068 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:42.141093 26068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:42.141357 26068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:42.143783 26068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "6ac4a581f0b9481297b56c6ac1ff9ec2"
format_stamp: "Formatted at 2026-09-26 15:19:42 on dist-test-slave-j6q7"
server_key: "948bd42a5cfde22fd40bb2a99e314e9b"
server_key_iv: "83ca3672bfcae002fa6bceeb6338d04c"
server_key_version: "encryptionkey@0"
I20260926 15:19:42.144423 26068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "6ac4a581f0b9481297b56c6ac1ff9ec2"
format_stamp: "Formatted at 2026-09-26 15:19:42 on dist-test-slave-j6q7"
server_key: "948bd42a5cfde22fd40bb2a99e314e9b"
server_key_iv: "83ca3672bfcae002fa6bceeb6338d04c"
server_key_version: "encryptionkey@0"
I20260926 15:19:42.148722 26068 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 15:19:42.151516 26082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:42.152654 26068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:19:42.152897 26068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "6ac4a581f0b9481297b56c6ac1ff9ec2"
format_stamp: "Formatted at 2026-09-26 15:19:42 on dist-test-slave-j6q7"
server_key: "948bd42a5cfde22fd40bb2a99e314e9b"
server_key_iv: "83ca3672bfcae002fa6bceeb6338d04c"
server_key_version: "encryptionkey@0"
I20260926 15:19:42.153072 26068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:42.175653 26068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:42.176290 26068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:42.176548 26068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:42.177354 26068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:19:42.178472 26068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:19:42.178580 26068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:42.178718 26068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:19:42.178807 26068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:42.208612 26068 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:44917
I20260926 15:19:42.208681 26194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:44917 every 8 connection(s)
I20260926 15:19:42.209726 26068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:19:42.216284 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 26068
I20260926 15:19:42.216446 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:19:42.216846 6983 external_mini_cluster.cc:1428] Setting key bea1fe0076d7c805fe219883b41b64b1
I20260926 15:19:42.220616 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:42.211920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:44055 (local address 127.6.209.254:40411)
0926 15:19:42.212084 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:42.212088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:42.213291 (+ 1203us) server_negotiation.cc:407] Connection header received
0926 15:19:42.214834 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:42.214839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:42.214921 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:42.215032 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:42.215898 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.216608 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.218191 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.218656 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.218740 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:42.220016 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:42.220021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:42.220024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:42.220040 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:42.220069 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:42.220080 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:42.220146 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:42.220335 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 15:19:42.220403 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 15:19:42.221460 26197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:42.212308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:40411 (local address 127.6.209.195:44055)
0926 15:19:42.213154 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:42.213188 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:42.214599 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:42.215218 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:42.215224 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:42.215303 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:19:42.215737 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:42.215749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.216743 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.216747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:42.218066 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:42.218075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.218848 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.218854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:42.219096 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:42.219749 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:19:42.219753 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:19:42.219784 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:19:42.219857 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:19:42.220160 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:19:42.220167 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:19:42.220172 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:19:42.220301 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 15:19:42.220536 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":237,"threads_started":1}
I20260926 15:19:42.222679 26195 heartbeater.cc:347] Connected to a master server at 127.6.209.254:40411
I20260926 15:19:42.222990 26195 heartbeater.cc:464] Registering TS with master...
I20260926 15:19:42.223547 26195 heartbeater.cc:511] Master 127.6.209.254:40411 requested a full tablet report, sending...
I20260926 15:19:42.224673 25749 ts_manager.cc:194] Registered new tserver with Master: 6ac4a581f0b9481297b56c6ac1ff9ec2 (127.6.209.195:44917)
I20260926 15:19:42.225420 25749 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.209.195:44055
I20260926 15:19:42.230934 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:19:42.239464 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:42.234620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47616 (local address 127.6.209.254:40411)
0926 15:19:42.234768 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:42.234771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:42.235186 (+ 415us) server_negotiation.cc:407] Connection header received
0926 15:19:42.235356 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:42.235359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:42.235410 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:42.235483 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:42.235863 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.236521 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.237716 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.238029 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.238106 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:42.238964 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:42.238968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:42.238970 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:42.238983 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:42.239007 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:42.239017 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:42.239103 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:42.239228 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 15:19:42.239268 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 15:19:42.242681 25749 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:19:42.248400 25749 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:19:42.271457 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:42.258568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:36285 (local address 127.0.0.1:51770)
0926 15:19:42.260626 (+ 2058us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:42.260647 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:42.260923 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:42.262047 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:42.262058 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:42.262096 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:42.262649 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:42.262663 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.264898 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.264905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:42.267759 (+ 2854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:42.267777 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.270410 (+ 2633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.270418 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:42.270667 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:42.270729 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 15:19:42.270854 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 15:19:42.271056 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1898,"thread_start_us":1747,"threads_started":1}
I20260926 15:19:42.272459 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:42.259409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51770 (local address 127.6.209.194:36285)
0926 15:19:42.260112 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:42.260123 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:42.260851 (+ 728us) server_negotiation.cc:407] Connection header received
0926 15:19:42.261141 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:42.261151 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:42.261440 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:42.261645 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:42.262906 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.264693 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.267982 (+ 3289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.269742 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.269910 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:42.270263 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:42.271745 (+ 1482us) server_negotiation.cc:299] Negotiation successful
0926 15:19:42.271978 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":326,"threads_started":1}
I20260926 15:19:42.276561 26207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:42.258339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:35013 (local address 127.0.0.1:57028)
0926 15:19:42.259285 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:42.259320 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:42.259471 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:42.270982 (+ 11511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:42.270986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:42.271010 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:42.271318 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:42.271326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.272672 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.272677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:42.274567 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:42.274583 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.276132 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.276140 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:42.276258 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:42.276276 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 15:19:42.276320 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 15:19:42.276384 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":615,"threads_started":1}
I20260926 15:19:42.276180 26000 tablet_service.cc:1467] Processing CreateTablet for tablet b39bc0c048bc40c3a74e2f78c5b5c7ed (DEFAULT_TABLE table=test-table [id=96ba142f55834fe69bccf186901ea96f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:19:42.277477 26000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39bc0c048bc40c3a74e2f78c5b5c7ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:42.277541 26208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:42.258483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57028 (local address 127.6.209.193:35013)
0926 15:19:42.270479 (+ 11996us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:42.270486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:42.270509 (+ 23us) server_negotiation.cc:407] Connection header received
0926 15:19:42.270589 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:42.270595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:42.270791 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:42.271023 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:42.271452 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.272497 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.274757 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.275943 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.276856 (+ 913us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:42.277045 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:42.277092 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 15:19:42.277218 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11854,"thread_start_us":11725,"threads_started":1}
I20260926 15:19:42.278975 26210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:42.259097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:44917 (local address 127.0.0.1:42472)
0926 15:19:42.269957 (+ 10860us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:42.269972 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:42.270059 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:42.273675 (+ 3616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:42.273679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:42.273699 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:42.274019 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:42.274027 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.275344 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.275349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:42.277079 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:42.277094 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.278392 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.278397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:42.278537 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:42.278571 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 15:19:42.278651 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 15:19:42.278748 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10708,"thread_start_us":10541,"threads_started":1}
I20260926 15:19:42.279145 26212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:42.259902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42472 (local address 127.6.209.195:44917)
0926 15:19:42.273293 (+ 13391us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:42.273298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:42.273319 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:19:42.273382 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:42.273386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:42.273507 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:42.273676 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:42.274177 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.275209 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.277264 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.278230 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.278367 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:42.278577 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:42.278696 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 15:19:42.278830 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13249,"thread_start_us":13101,"threads_started":1}
I20260926 15:19:42.280647 25870 tablet_service.cc:1467] Processing CreateTablet for tablet b39bc0c048bc40c3a74e2f78c5b5c7ed (DEFAULT_TABLE table=test-table [id=96ba142f55834fe69bccf186901ea96f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:19:42.282045 25870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39bc0c048bc40c3a74e2f78c5b5c7ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:42.281906 26130 tablet_service.cc:1467] Processing CreateTablet for tablet b39bc0c048bc40c3a74e2f78c5b5c7ed (DEFAULT_TABLE table=test-table [id=96ba142f55834fe69bccf186901ea96f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:19:42.283002 26130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39bc0c048bc40c3a74e2f78c5b5c7ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:42.293573 26213 tablet_bootstrap.cc:492] T b39bc0c048bc40c3a74e2f78c5b5c7ed P d09fc6da80214d32bf8b39f2bf5d6c25: Bootstrap starting.
I20260926 15:19:42.295228 26215 tablet_bootstrap.cc:492] T b39bc0c048bc40c3a74e2f78c5b5c7ed P aadff1e71c214e289335453ad94b7758: Bootstrap starting.
I20260926 15:19:42.297668 26213 tablet_bootstrap.cc:654] T b39bc0c048bc40c3a74e2f78c5b5c7ed P d09fc6da80214d32bf8b39f2bf5d6c25: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:42.298004 26215 tablet_bootstrap.cc:654] T b39bc0c048bc40c3a74e2f78c5b5c7ed P aadff1e71c214e289335453ad94b7758: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:42.299005 26215 log.cc:824] T b39bc0c048bc40c3a74e2f78c5b5c7ed P aadff1e71c214e289335453ad94b7758: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:42.299045 26213 log.cc:824] T b39bc0c048bc40c3a74e2f78c5b5c7ed P d09fc6da80214d32bf8b39f2bf5d6c25: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:42.299716 26214 tablet_bootstrap.cc:492] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2: Bootstrap starting.
I20260926 15:19:42.301939 26215 tablet_bootstrap.cc:492] T b39bc0c048bc40c3a74e2f78c5b5c7ed P aadff1e71c214e289335453ad94b7758: No bootstrap required, opened a new log
I20260926 15:19:42.302187 26215 ts_tablet_manager.cc:1397] T b39bc0c048bc40c3a74e2f78c5b5c7ed P aadff1e71c214e289335453ad94b7758: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 15:19:42.302763 26213 tablet_bootstrap.cc:492] T b39bc0c048bc40c3a74e2f78c5b5c7ed P d09fc6da80214d32bf8b39f2bf5d6c25: No bootstrap required, opened a new log
I20260926 15:19:42.302973 26214 tablet_bootstrap.cc:654] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:42.303007 26213 ts_tablet_manager.cc:1397] T b39bc0c048bc40c3a74e2f78c5b5c7ed P d09fc6da80214d32bf8b39f2bf5d6c25: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260926 15:19:42.304105 26214 log.cc:824] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:42.306751 26214 tablet_bootstrap.cc:492] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2: No bootstrap required, opened a new log
I20260926 15:19:42.306964 26214 ts_tablet_manager.cc:1397] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 15:19:42.309326 26215 raft_consensus.cc:348] T b39bc0c048bc40c3a74e2f78c5b5c7ed P aadff1e71c214e289335453ad94b7758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aadff1e71c214e289335453ad94b7758" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 35013 } } peers { permanent_uuid: "d09fc6da80214d32bf8b39f2bf5d6c25" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36285 } } peers { permanent_uuid: "6ac4a581f0b9481297b56c6ac1ff9ec2" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44917 } }
I20260926 15:19:42.309623 26215 raft_consensus.cc:374] T b39bc0c048bc40c3a74e2f78c5b5c7ed P aadff1e71c214e289335453ad94b7758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:42.309727 26215 raft_consensus.cc:729] T b39bc0c048bc40c3a74e2f78c5b5c7ed P aadff1e71c214e289335453ad94b7758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aadff1e71c214e289335453ad94b7758, State: Initialized, Role: FOLLOWER
I20260926 15:19:42.310278 26215 consensus_queue.cc:260] T b39bc0c048bc40c3a74e2f78c5b5c7ed P aadff1e71c214e289335453ad94b7758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aadff1e71c214e289335453ad94b7758" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 35013 } } peers { permanent_uuid: "d09fc6da80214d32bf8b39f2bf5d6c25" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36285 } } peers { permanent_uuid: "6ac4a581f0b9481297b56c6ac1ff9ec2" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44917 } }
I20260926 15:19:42.311214 25935 heartbeater.cc:503] Master 127.6.209.254:40411 was elected leader, sending a full tablet report...
I20260926 15:19:42.311836 26215 ts_tablet_manager.cc:1428] T b39bc0c048bc40c3a74e2f78c5b5c7ed P aadff1e71c214e289335453ad94b7758: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260926 15:19:42.312229 26213 raft_consensus.cc:348] T b39bc0c048bc40c3a74e2f78c5b5c7ed P d09fc6da80214d32bf8b39f2bf5d6c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aadff1e71c214e289335453ad94b7758" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 35013 } } peers { permanent_uuid: "d09fc6da80214d32bf8b39f2bf5d6c25" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36285 } } peers { permanent_uuid: "6ac4a581f0b9481297b56c6ac1ff9ec2" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44917 } }
I20260926 15:19:42.312621 26213 raft_consensus.cc:374] T b39bc0c048bc40c3a74e2f78c5b5c7ed P d09fc6da80214d32bf8b39f2bf5d6c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:42.312732 26213 raft_consensus.cc:729] T b39bc0c048bc40c3a74e2f78c5b5c7ed P d09fc6da80214d32bf8b39f2bf5d6c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09fc6da80214d32bf8b39f2bf5d6c25, State: Initialized, Role: FOLLOWER
I20260926 15:19:42.313251 26213 consensus_queue.cc:260] T b39bc0c048bc40c3a74e2f78c5b5c7ed P d09fc6da80214d32bf8b39f2bf5d6c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aadff1e71c214e289335453ad94b7758" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 35013 } } peers { permanent_uuid: "d09fc6da80214d32bf8b39f2bf5d6c25" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36285 } } peers { permanent_uuid: "6ac4a581f0b9481297b56c6ac1ff9ec2" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44917 } }
I20260926 15:19:42.314208 26065 heartbeater.cc:503] Master 127.6.209.254:40411 was elected leader, sending a full tablet report...
I20260926 15:19:42.314694 26213 ts_tablet_manager.cc:1428] T b39bc0c048bc40c3a74e2f78c5b5c7ed P d09fc6da80214d32bf8b39f2bf5d6c25: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260926 15:19:42.316107 26214 raft_consensus.cc:348] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aadff1e71c214e289335453ad94b7758" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 35013 } } peers { permanent_uuid: "d09fc6da80214d32bf8b39f2bf5d6c25" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36285 } } peers { permanent_uuid: "6ac4a581f0b9481297b56c6ac1ff9ec2" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44917 } }
I20260926 15:19:42.316452 26214 raft_consensus.cc:374] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:42.316542 26214 raft_consensus.cc:729] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ac4a581f0b9481297b56c6ac1ff9ec2, State: Initialized, Role: FOLLOWER
I20260926 15:19:42.317008 26214 consensus_queue.cc:260] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aadff1e71c214e289335453ad94b7758" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 35013 } } peers { permanent_uuid: "d09fc6da80214d32bf8b39f2bf5d6c25" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36285 } } peers { permanent_uuid: "6ac4a581f0b9481297b56c6ac1ff9ec2" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44917 } }
I20260926 15:19:42.317905 26195 heartbeater.cc:503] Master 127.6.209.254:40411 was elected leader, sending a full tablet report...
I20260926 15:19:42.318295 26214 ts_tablet_manager.cc:1428] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260926 15:19:42.325783 26221 raft_consensus.cc:482] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:19:42.325960 26221 raft_consensus.cc:504] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aadff1e71c214e289335453ad94b7758" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 35013 } } peers { permanent_uuid: "d09fc6da80214d32bf8b39f2bf5d6c25" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36285 } } peers { permanent_uuid: "6ac4a581f0b9481297b56c6ac1ff9ec2" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44917 } }
I20260926 15:19:42.327137 26221 leader_election.cc:288] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aadff1e71c214e289335453ad94b7758 (127.6.209.193:35013), d09fc6da80214d32bf8b39f2bf5d6c25 (127.6.209.194:36285)
I20260926 15:19:42.334051 26208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:42.327576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:47713 (local address 127.6.209.193:35013)
0926 15:19:42.327731 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:42.327736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:42.327805 (+ 69us) server_negotiation.cc:407] Connection header received
0926 15:19:42.327991 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:42.327998 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:42.328071 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:42.328162 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:42.328621 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.329649 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.331942 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.333166 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.333327 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:42.333405 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:42.333688 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 15:19:42.333821 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 15:19:42.334056 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:42.327502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:57439 (local address 127.6.209.194:36285)
0926 15:19:42.327647 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:42.327653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:42.328286 (+ 633us) server_negotiation.cc:407] Connection header received
0926 15:19:42.328407 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:42.328410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:42.328453 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:42.328510 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:42.329033 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.329931 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.332087 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.333296 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.333422 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:42.333487 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:42.333733 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 15:19:42.333850 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 15:19:42.334136 26197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:42.327509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:35013 (local address 127.6.209.195:47713)
0926 15:19:42.327725 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:42.327737 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:42.327860 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:42.328204 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:42.328210 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:42.328238 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:42.328455 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:42.328462 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.329762 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.329765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:42.331764 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:42.331778 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.333317 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.333321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:42.333524 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:42.333558 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 15:19:42.333651 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 15:19:42.333850 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":99,"spinlock_wait_cycles":14592}
I20260926 15:19:42.334273 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:42.327605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:36285 (local address 127.6.209.195:57439)
0926 15:19:42.328181 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:42.328198 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:42.328310 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:42.328546 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:42.328549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:42.328571 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:42.328894 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:42.328902 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.330029 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.330033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:42.331935 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:42.331945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.333432 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.333437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:42.333586 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:42.333626 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 15:19:42.333698 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 15:19:42.333850 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":28,"thread_start_us":317,"threads_started":1}
I20260926 15:19:42.334945 26020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39bc0c048bc40c3a74e2f78c5b5c7ed" candidate_uuid: "6ac4a581f0b9481297b56c6ac1ff9ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09fc6da80214d32bf8b39f2bf5d6c25" is_pre_election: true
I20260926 15:19:42.335059 25890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39bc0c048bc40c3a74e2f78c5b5c7ed" candidate_uuid: "6ac4a581f0b9481297b56c6ac1ff9ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aadff1e71c214e289335453ad94b7758" is_pre_election: true
I20260926 15:19:42.335382 26020 raft_consensus.cc:2454] T b39bc0c048bc40c3a74e2f78c5b5c7ed P d09fc6da80214d32bf8b39f2bf5d6c25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ac4a581f0b9481297b56c6ac1ff9ec2 in term 0.
I20260926 15:19:42.335583 25890 raft_consensus.cc:2454] T b39bc0c048bc40c3a74e2f78c5b5c7ed P aadff1e71c214e289335453ad94b7758 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ac4a581f0b9481297b56c6ac1ff9ec2 in term 0.
I20260926 15:19:42.335886 26084 leader_election.cc:302] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ac4a581f0b9481297b56c6ac1ff9ec2, d09fc6da80214d32bf8b39f2bf5d6c25; no voters:
I20260926 15:19:42.336256 26221 raft_consensus.cc:2781] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:19:42.336404 26221 raft_consensus.cc:482] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:19:42.336490 26221 raft_consensus.cc:3037] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:42.338347 26221 raft_consensus.cc:504] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aadff1e71c214e289335453ad94b7758" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 35013 } } peers { permanent_uuid: "d09fc6da80214d32bf8b39f2bf5d6c25" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36285 } } peers { permanent_uuid: "6ac4a581f0b9481297b56c6ac1ff9ec2" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44917 } }
I20260926 15:19:42.338814 26221 leader_election.cc:288] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2 [CANDIDATE]: Term 1 election: Requested vote from peers aadff1e71c214e289335453ad94b7758 (127.6.209.193:35013), d09fc6da80214d32bf8b39f2bf5d6c25 (127.6.209.194:36285)
I20260926 15:19:42.339179 25890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39bc0c048bc40c3a74e2f78c5b5c7ed" candidate_uuid: "6ac4a581f0b9481297b56c6ac1ff9ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aadff1e71c214e289335453ad94b7758"
I20260926 15:19:42.339195 26020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39bc0c048bc40c3a74e2f78c5b5c7ed" candidate_uuid: "6ac4a581f0b9481297b56c6ac1ff9ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09fc6da80214d32bf8b39f2bf5d6c25"
I20260926 15:19:42.339349 25890 raft_consensus.cc:3037] T b39bc0c048bc40c3a74e2f78c5b5c7ed P aadff1e71c214e289335453ad94b7758 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:42.339355 26020 raft_consensus.cc:3037] T b39bc0c048bc40c3a74e2f78c5b5c7ed P d09fc6da80214d32bf8b39f2bf5d6c25 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:42.341473 26020 raft_consensus.cc:2454] T b39bc0c048bc40c3a74e2f78c5b5c7ed P d09fc6da80214d32bf8b39f2bf5d6c25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ac4a581f0b9481297b56c6ac1ff9ec2 in term 1.
I20260926 15:19:42.341527 25890 raft_consensus.cc:2454] T b39bc0c048bc40c3a74e2f78c5b5c7ed P aadff1e71c214e289335453ad94b7758 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ac4a581f0b9481297b56c6ac1ff9ec2 in term 1.
I20260926 15:19:42.341913 26084 leader_election.cc:302] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ac4a581f0b9481297b56c6ac1ff9ec2, d09fc6da80214d32bf8b39f2bf5d6c25; no voters:
I20260926 15:19:42.342175 26221 raft_consensus.cc:2781] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:19:42.342523 26221 raft_consensus.cc:686] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2 [term 1 LEADER]: Becoming Leader. State: Replica: 6ac4a581f0b9481297b56c6ac1ff9ec2, State: Running, Role: LEADER
I20260926 15:19:42.343012 26221 consensus_queue.cc:237] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aadff1e71c214e289335453ad94b7758" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 35013 } } peers { permanent_uuid: "d09fc6da80214d32bf8b39f2bf5d6c25" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36285 } } peers { permanent_uuid: "6ac4a581f0b9481297b56c6ac1ff9ec2" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44917 } }
I20260926 15:19:42.346450 25748 catalog_manager.cc:5495] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ac4a581f0b9481297b56c6ac1ff9ec2 (127.6.209.195). New cstate: current_term: 1 leader_uuid: "6ac4a581f0b9481297b56c6ac1ff9ec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aadff1e71c214e289335453ad94b7758" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 35013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d09fc6da80214d32bf8b39f2bf5d6c25" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ac4a581f0b9481297b56c6ac1ff9ec2" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 44917 } health_report { overall_health: HEALTHY } } }
I20260926 15:19:42.365993 26212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:42.361227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42482 (local address 127.6.209.195:44917)
0926 15:19:42.361406 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:42.361413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:42.361522 (+ 109us) server_negotiation.cc:407] Connection header received
0926 15:19:42.361693 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:42.361696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:42.361767 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:42.361876 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:19:42.362218 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.362912 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.364090 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.364449 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.364566 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:42.365144 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:19:42.365520 (+ 376us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:19:42.365684 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:42.365785 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 15:19:42.365820 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 15:19:42.372638 26020 raft_consensus.cc:1261] T b39bc0c048bc40c3a74e2f78c5b5c7ed P d09fc6da80214d32bf8b39f2bf5d6c25 [term 1 FOLLOWER]: Refusing update from remote peer 6ac4a581f0b9481297b56c6ac1ff9ec2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:19:42.372674 25890 raft_consensus.cc:1261] T b39bc0c048bc40c3a74e2f78c5b5c7ed P aadff1e71c214e289335453ad94b7758 [term 1 FOLLOWER]: Refusing update from remote peer 6ac4a581f0b9481297b56c6ac1ff9ec2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:19:42.373456 26221 consensus_queue.cc:1035] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aadff1e71c214e289335453ad94b7758" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 35013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:19:42.373701 26223 consensus_queue.cc:1035] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09fc6da80214d32bf8b39f2bf5d6c25" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 36285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:19:42.386107 26227 mvcc.cc:205] Tried to move back new op lower bound from 7333625783790534656 to 7333625783680126976. Current Snapshot: MvccSnapshot[applied={T|T < 7333625783790534656}]
I20260926 15:19:42.396385 25748 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:19:42.396633 25748 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:19:42.399822 25748 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4789cf05bafd4e8fb8c705df4a91728a: Sending DeleteTablet for 3 replicas of tablet b39bc0c048bc40c3a74e2f78c5b5c7ed
I20260926 15:19:42.400879 25870 tablet_service.cc:1514] Processing DeleteTablet for tablet b39bc0c048bc40c3a74e2f78c5b5c7ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:19:42 UTC) from {username='slave'} at 127.0.0.1:57028
I20260926 15:19:42.400888 26000 tablet_service.cc:1514] Processing DeleteTablet for tablet b39bc0c048bc40c3a74e2f78c5b5c7ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:19:42 UTC) from {username='slave'} at 127.0.0.1:51770
I20260926 15:19:42.401664 26236 tablet_replica.cc:330] T b39bc0c048bc40c3a74e2f78c5b5c7ed P d09fc6da80214d32bf8b39f2bf5d6c25: stopping tablet replica
I20260926 15:19:42.401718 26130 tablet_service.cc:1514] Processing DeleteTablet for tablet b39bc0c048bc40c3a74e2f78c5b5c7ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:19:42 UTC) from {username='slave'} at 127.0.0.1:42472
I20260926 15:19:42.402055 26236 raft_consensus.cc:2229] T b39bc0c048bc40c3a74e2f78c5b5c7ed P d09fc6da80214d32bf8b39f2bf5d6c25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:19:42.403354 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 25808
I20260926 15:19:42.411182 26238 tablet_replica.cc:330] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2: stopping tablet replica
I20260926 15:19:42.411794 26238 raft_consensus.cc:2229] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:19:42.412261 26236 raft_consensus.cc:2258] T b39bc0c048bc40c3a74e2f78c5b5c7ed P d09fc6da80214d32bf8b39f2bf5d6c25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:19:42.412892 26238 raft_consensus.cc:2258] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:19:42.413549 26237 tablet_replica.cc:330] T b39bc0c048bc40c3a74e2f78c5b5c7ed P aadff1e71c214e289335453ad94b7758: stopping tablet replica
I20260926 15:19:42.413935 26237 raft_consensus.cc:2229] T b39bc0c048bc40c3a74e2f78c5b5c7ed P aadff1e71c214e289335453ad94b7758 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:19:42.414368 26237 raft_consensus.cc:2258] T b39bc0c048bc40c3a74e2f78c5b5c7ed P aadff1e71c214e289335453ad94b7758 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:19:42.415407 26238 ts_tablet_manager.cc:1905] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:19:42.415856 26236 ts_tablet_manager.cc:1905] T b39bc0c048bc40c3a74e2f78c5b5c7ed P d09fc6da80214d32bf8b39f2bf5d6c25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:19:42.416025 26237 ts_tablet_manager.cc:1905] T b39bc0c048bc40c3a74e2f78c5b5c7ed P aadff1e71c214e289335453ad94b7758: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:19:42.422112 26238 ts_tablet_manager.cc:1918] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:19:42.422325 26238 log.cc:1196] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/b39bc0c048bc40c3a74e2f78c5b5c7ed
I20260926 15:19:42.422870 26237 ts_tablet_manager.cc:1918] T b39bc0c048bc40c3a74e2f78c5b5c7ed P aadff1e71c214e289335453ad94b7758: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:19:42.422895 26238 ts_tablet_manager.cc:1939] T b39bc0c048bc40c3a74e2f78c5b5c7ed P 6ac4a581f0b9481297b56c6ac1ff9ec2: Deleting consensus metadata
I20260926 15:19:42.423053 26237 log.cc:1196] T b39bc0c048bc40c3a74e2f78c5b5c7ed P aadff1e71c214e289335453ad94b7758: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-0/wal/wals/b39bc0c048bc40c3a74e2f78c5b5c7ed
I20260926 15:19:42.423523 26237 ts_tablet_manager.cc:1939] T b39bc0c048bc40c3a74e2f78c5b5c7ed P aadff1e71c214e289335453ad94b7758: Deleting consensus metadata
I20260926 15:19:42.424299 25735 catalog_manager.cc:4841] TS 6ac4a581f0b9481297b56c6ac1ff9ec2 (127.6.209.195:44917): tablet b39bc0c048bc40c3a74e2f78c5b5c7ed (table test-table [id=96ba142f55834fe69bccf186901ea96f]) successfully deleted
I20260926 15:19:42.424841 25733 catalog_manager.cc:4841] TS aadff1e71c214e289335453ad94b7758 (127.6.209.193:35013): tablet b39bc0c048bc40c3a74e2f78c5b5c7ed (table test-table [id=96ba142f55834fe69bccf186901ea96f]) successfully deleted
I20260926 15:19:42.431149 26236 ts_tablet_manager.cc:1918] T b39bc0c048bc40c3a74e2f78c5b5c7ed P d09fc6da80214d32bf8b39f2bf5d6c25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:19:42.431313 26236 log.cc:1196] T b39bc0c048bc40c3a74e2f78c5b5c7ed P d09fc6da80214d32bf8b39f2bf5d6c25: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/b39bc0c048bc40c3a74e2f78c5b5c7ed
I20260926 15:19:42.431844 26236 ts_tablet_manager.cc:1939] T b39bc0c048bc40c3a74e2f78c5b5c7ed P d09fc6da80214d32bf8b39f2bf5d6c25: Deleting consensus metadata
I20260926 15:19:42.433460 25734 catalog_manager.cc:4841] TS d09fc6da80214d32bf8b39f2bf5d6c25 (127.6.209.194:36285): tablet b39bc0c048bc40c3a74e2f78c5b5c7ed (table test-table [id=96ba142f55834fe69bccf186901ea96f]) successfully deleted
I20260926 15:19:42.436038 26208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:42.406267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57038 (local address 127.6.209.193:35013)
0926 15:19:42.406411 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:42.406416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:42.429311 (+ 22895us) server_negotiation.cc:407] Connection header received
0926 15:19:42.429502 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:42.429507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:42.429579 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:42.429638 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:42.430167 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.431170 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.433108 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.433647 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.433730 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:42.434786 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:42.434794 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:42.434806 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:42.435355 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:42.435400 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:42.435425 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:42.435511 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:42.435654 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 15:19:42.435738 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 15:19:42.436511 25930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:42.517279 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 25808
I20260926 15:19:42.534078 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 25938
I20260926 15:19:42.540798 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:42.534852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.6.209.194:36285)
0926 15:19:42.535020 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:42.535027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:42.535047 (+ 20us) server_negotiation.cc:407] Connection header received
0926 15:19:42.535122 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:42.535128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:42.535230 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:42.535340 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:42.535716 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.536703 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.537785 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.538335 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.538447 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:42.539350 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:42.539360 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:42.539378 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:42.540107 (+ 729us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:42.540166 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:42.540191 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:42.540313 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:42.540367 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 15:19:42.540468 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 15:19:42.541440 26060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:42.624680 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 25938
I20260926 15:19:42.646279 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 26068
I20260926 15:19:42.654062 26212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:42.647619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42490 (local address 127.6.209.195:44917)
0926 15:19:42.647941 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:42.647947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:42.647966 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:19:42.648030 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:42.648035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:42.648128 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:42.648268 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:42.648799 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.649886 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.650997 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:42.651405 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:42.651529 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:42.652640 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:42.652647 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:42.652661 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:42.653427 (+ 766us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:42.653474 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:42.653492 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:42.653598 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:42.653688 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 15:19:42.653765 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 15:19:42.654886 26190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:42.763146 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 26068
I20260926 15:19:42.781442 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 25718
I20260926 15:19:42.782471 25779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:42.878376 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 25718
2026-09-26T15:19:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2411 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:19:42 dist-test-slave-j6q7 krb5kdc[26246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:19:42 dist-test-slave-j6q7 krb5kdc[26246](info): set up 2 sockets
Sep 26 15:19:42 dist-test-slave-j6q7 krb5kdc[26246](info): commencing operation
krb5kdc: starting...
W20260926 15:19:43.896627 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:19:44 dist-test-slave-j6q7 krb5kdc[26246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:19:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:19:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:19:44.091768 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:42571
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:39375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:42571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:19:44.381120 26259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:44.381367 26259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:44.381449 26259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:44.388795 26259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:19:44.388921 26259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:44.389001 26259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:44.389081 26259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:19:44.389142 26259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:19:44.397841 26259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:39375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:42571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:42571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:44.398867 26259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:44.400279 26259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:44.411542 26264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:44.411459 26265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:44.411456 26267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:44.412158 26259 server_base.cc:1034] running on GCE node
I20260926 15:19:44.412976 26259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:44.414253 26259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:44.415478 26259 hybrid_clock.cc:648] HybridClock initialized: now 1790435984415443 us; error 43 us; skew 500 ppm
I20260926 15:19:44.415823 26259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:19:44 dist-test-slave-j6q7 krb5kdc[26246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435984, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:19:44.421779 26259 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:19:44.423233 26259 webserver.cc:460] Webserver started at http://127.6.209.254:42591/ using document root <none> and password file <none>
I20260926 15:19:44.423856 26259 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:44.423990 26259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:44.424276 26259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:44.426991 26259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "9c1e7db9f2424ee4928e7675ab209254"
format_stamp: "Formatted at 2026-09-26 15:19:44 on dist-test-slave-j6q7"
server_key: "b24f63ada173827b802c5a76b37681a1"
server_key_iv: "f762c737cb7740c302f83761765a7a1c"
server_key_version: "encryptionkey@0"
I20260926 15:19:44.427690 26259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "9c1e7db9f2424ee4928e7675ab209254"
format_stamp: "Formatted at 2026-09-26 15:19:44 on dist-test-slave-j6q7"
server_key: "b24f63ada173827b802c5a76b37681a1"
server_key_iv: "f762c737cb7740c302f83761765a7a1c"
server_key_version: "encryptionkey@0"
I20260926 15:19:44.432037 26259 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 15:19:44.435123 26274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:44.436244 26259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 15:19:44.436407 26259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "9c1e7db9f2424ee4928e7675ab209254"
format_stamp: "Formatted at 2026-09-26 15:19:44 on dist-test-slave-j6q7"
server_key: "b24f63ada173827b802c5a76b37681a1"
server_key_iv: "f762c737cb7740c302f83761765a7a1c"
server_key_version: "encryptionkey@0"
I20260926 15:19:44.436570 26259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:44.454124 26259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:44.457785 26259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:44.458021 26259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:44.474557 26259 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:42571
I20260926 15:19:44.474587 26325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:42571 every 8 connection(s)
I20260926 15:19:44.475708 26259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:19:44.478754 26326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:44.479396 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 26259
I20260926 15:19:44.479575 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:19:44.480055 6983 external_mini_cluster.cc:1428] Setting key 986549878b59a851aa06705c995cab8b
I20260926 15:19:44.492057 26326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c1e7db9f2424ee4928e7675ab209254: Bootstrap starting.
Sep 26 15:19:44 dist-test-slave-j6q7 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:19:44.494840 26326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c1e7db9f2424ee4928e7675ab209254: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:44.496059 26326 log.cc:824] T 00000000000000000000000000000000 P 9c1e7db9f2424ee4928e7675ab209254: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:44.499007 26326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c1e7db9f2424ee4928e7675ab209254: No bootstrap required, opened a new log
I20260926 15:19:44.502614 26329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:44.481719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44878 (local address 127.6.209.254:42571)
0926 15:19:44.482493 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:44.482507 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:44.482556 (+ 49us) server_negotiation.cc:407] Connection header received
0926 15:19:44.485083 (+ 2527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:44.485108 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:44.485561 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:44.485965 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:44.487158 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:44.489001 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:44.490209 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:44.491140 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:44.491312 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:44.494943 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:44.494952 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:44.494968 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:44.495021 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:44.498498 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:44.499104 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:44.499110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:44.499117 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:44.499304 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:44.499644 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:44.499649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:44.499652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:44.500339 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:44.500625 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:44.500960 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 15:19:44.501374 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":309,"threads_started":1}
I20260926 15:19:44.507606 26326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c1e7db9f2424ee4928e7675ab209254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c1e7db9f2424ee4928e7675ab209254" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 42571 } }
I20260926 15:19:44.507911 26326 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c1e7db9f2424ee4928e7675ab209254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:44.507980 26326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c1e7db9f2424ee4928e7675ab209254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c1e7db9f2424ee4928e7675ab209254, State: Initialized, Role: FOLLOWER
I20260926 15:19:44.508399 26326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c1e7db9f2424ee4928e7675ab209254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c1e7db9f2424ee4928e7675ab209254" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 42571 } }
I20260926 15:19:44.508574 26326 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c1e7db9f2424ee4928e7675ab209254 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:19:44.508656 26326 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c1e7db9f2424ee4928e7675ab209254 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:19:44.508770 26326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c1e7db9f2424ee4928e7675ab209254 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:44.510583 26326 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c1e7db9f2424ee4928e7675ab209254 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c1e7db9f2424ee4928e7675ab209254" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 42571 } }
I20260926 15:19:44.511087 26326 leader_election.cc:302] T 00000000000000000000000000000000 P 9c1e7db9f2424ee4928e7675ab209254 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c1e7db9f2424ee4928e7675ab209254; no voters:
I20260926 15:19:44.511673 26326 leader_election.cc:288] T 00000000000000000000000000000000 P 9c1e7db9f2424ee4928e7675ab209254 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:19:44.511751 26331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c1e7db9f2424ee4928e7675ab209254 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:19:44.512447 26331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c1e7db9f2424ee4928e7675ab209254 [term 1 LEADER]: Becoming Leader. State: Replica: 9c1e7db9f2424ee4928e7675ab209254, State: Running, Role: LEADER
I20260926 15:19:44.513031 26331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c1e7db9f2424ee4928e7675ab209254 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c1e7db9f2424ee4928e7675ab209254" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 42571 } }
I20260926 15:19:44.513651 26326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c1e7db9f2424ee4928e7675ab209254 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:19:44.515286 26332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c1e7db9f2424ee4928e7675ab209254 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c1e7db9f2424ee4928e7675ab209254" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c1e7db9f2424ee4928e7675ab209254" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 42571 } } }
I20260926 15:19:44.515473 26333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c1e7db9f2424ee4928e7675ab209254 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c1e7db9f2424ee4928e7675ab209254. Latest consensus state: current_term: 1 leader_uuid: "9c1e7db9f2424ee4928e7675ab209254" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c1e7db9f2424ee4928e7675ab209254" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 42571 } } }
I20260926 15:19:44.515697 26333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c1e7db9f2424ee4928e7675ab209254 [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:44.515709 26332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c1e7db9f2424ee4928e7675ab209254 [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:44.521109 26337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:19:44.525789 26337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:19:44.534837 26337 catalog_manager.cc:1347] Generated new cluster ID: 9db7a17a6c68409284026c3ec0bc9101
I20260926 15:19:44.534940 26337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:19:44.553498 26337 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:19:44.554538 26337 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:19:44.564905 26337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c1e7db9f2424ee4928e7675ab209254: Generated new TSK 0
I20260926 15:19:44.565511 26337 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
Entry for principal kudu/127.6.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab.
I20260926 15:19:44.622315 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:42571
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:39375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:19:44.920163 26352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:44.920397 26352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:44.920547 26352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:44.927335 26352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:44.927414 26352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:44.927579 26352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:19:44.936695 26352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:39375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.193.keytab
--principal=kudu/127.6.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:42571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:44.937727 26352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:44.938891 26352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:44.950641 26357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:44.950634 26360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:44.950659 26358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:44.951476 26352 server_base.cc:1034] running on GCE node
I20260926 15:19:44.951996 26352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:44.952895 26352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:44.954106 26352 hybrid_clock.cc:648] HybridClock initialized: now 1790435984954073 us; error 42 us; skew 500 ppm
I20260926 15:19:44.954394 26352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:19:44 dist-test-slave-j6q7 krb5kdc[26246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435984, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:19:44.957150 26352 init.cc:377] Logged in from keytab as kudu/127.6.209.193@KRBTEST.COM (short username kudu)
I20260926 15:19:44.958352 26352 webserver.cc:460] Webserver started at http://127.6.209.193:45373/ using document root <none> and password file <none>
I20260926 15:19:44.958933 26352 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:44.959056 26352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:44.959324 26352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:44.962026 26352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "6f8a6f4b3d214315a9addd6adeacba8f"
format_stamp: "Formatted at 2026-09-26 15:19:44 on dist-test-slave-j6q7"
server_key: "f279c683feff9ee2e15ac5ca9e252065"
server_key_iv: "30e73569885c6403227f9e80134e1ce6"
server_key_version: "encryptionkey@0"
I20260926 15:19:44.962781 26352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "6f8a6f4b3d214315a9addd6adeacba8f"
format_stamp: "Formatted at 2026-09-26 15:19:44 on dist-test-slave-j6q7"
server_key: "f279c683feff9ee2e15ac5ca9e252065"
server_key_iv: "30e73569885c6403227f9e80134e1ce6"
server_key_version: "encryptionkey@0"
I20260926 15:19:44.967036 26352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 15:19:44.969877 26367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:44.970937 26352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:19:44.971122 26352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "6f8a6f4b3d214315a9addd6adeacba8f"
format_stamp: "Formatted at 2026-09-26 15:19:44 on dist-test-slave-j6q7"
server_key: "f279c683feff9ee2e15ac5ca9e252065"
server_key_iv: "30e73569885c6403227f9e80134e1ce6"
server_key_version: "encryptionkey@0"
I20260926 15:19:44.971299 26352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:44.987735 26352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:44.990940 26352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:44.991187 26352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:44.992031 26352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:19:44.993249 26352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:19:44.993323 26352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:44.993417 26352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:19:44.993456 26352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:45.023612 26352 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:32835
I20260926 15:19:45.023654 26479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:32835 every 8 connection(s)
I20260926 15:19:45.024740 26352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:19:45.032063 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 26352
I20260926 15:19:45.032212 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:19:45.032629 6983 external_mini_cluster.cc:1428] Setting key d853eca9d4d5b4c8cb70efe0b40f0a4f
Sep 26 15:19:45 dist-test-slave-j6q7 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435984, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.193@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:19:45.043854 26482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:45.027146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:42717 (local address 127.6.209.254:42571)
0926 15:19:45.027576 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:45.027582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:45.028444 (+ 862us) server_negotiation.cc:407] Connection header received
0926 15:19:45.030098 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:45.030103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:45.030185 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:45.030295 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:45.031634 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:45.032491 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:45.034085 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:45.034427 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:45.034549 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:45.038036 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:45.038041 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:45.038044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:45.038072 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:45.040628 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:45.041513 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:45.041521 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:45.041526 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:45.041670 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:45.042063 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:45.042069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:45.042073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:45.042464 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:45.042622 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:45.043223 (+ 601us) server_negotiation.cc:299] Negotiation successful
0926 15:19:45.043445 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":197,"threads_started":1}
I20260926 15:19:45.044292 26483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:45.027703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:42571 (local address 127.6.209.193:42717)
0926 15:19:45.028309 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:45.028345 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:45.029844 (+ 1499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:45.030511 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:45.030518 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:45.030968 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:19:45.031450 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:45.031465 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:45.032647 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:45.032651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:45.033950 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:45.033961 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:45.034627 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:45.034633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:45.034854 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:45.035486 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:19:45.035512 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:19:45.037853 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:19:45.040749 (+ 2896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:19:45.040759 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:19:45.040775 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:19:45.041387 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:19:45.041770 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:19:45.041773 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:19:45.041775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:19:45.041941 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:19:45.042582 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:19:45.042587 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:19:45.042851 (+ 264us) client_negotiation.cc:769] Sending connection context
0926 15:19:45.043072 (+ 221us) client_negotiation.cc:240] Negotiation successful
0926 15:19:45.043385 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":226,"threads_started":1}
I20260926 15:19:45.045537 26480 heartbeater.cc:347] Connected to a master server at 127.6.209.254:42571
I20260926 15:19:45.045782 26480 heartbeater.cc:464] Registering TS with master...
I20260926 15:19:45.046365 26480 heartbeater.cc:511] Master 127.6.209.254:42571 requested a full tablet report, sending...
I20260926 15:19:45.048110 26291 ts_manager.cc:194] Registered new tserver with Master: 6f8a6f4b3d214315a9addd6adeacba8f (127.6.209.193:32835)
I20260926 15:19:45.049387 26291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.193@KRBTEST.COM'} at 127.6.209.193:42717
WARNING: no policy specified for kudu/127.6.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
Entry for principal kudu/127.6.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab.
I20260926 15:19:45.080139 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:42571
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:39375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:19:45.372164 26486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:45.372416 26486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:45.372620 26486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:45.379134 26486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:45.379216 26486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:45.379309 26486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:19:45.390427 26486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:39375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.194.keytab
--principal=kudu/127.6.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:42571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:45.391822 26486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:45.393548 26486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:45.405869 26491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:45.405884 26492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:45.405912 26494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:45.406569 26486 server_base.cc:1034] running on GCE node
I20260926 15:19:45.407100 26486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:45.407971 26486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:45.409181 26486 hybrid_clock.cc:648] HybridClock initialized: now 1790435985409131 us; error 42 us; skew 500 ppm
I20260926 15:19:45.409426 26486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:19:45 dist-test-slave-j6q7 krb5kdc[26246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435985, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:19:45.412415 26486 init.cc:377] Logged in from keytab as kudu/127.6.209.194@KRBTEST.COM (short username kudu)
I20260926 15:19:45.413858 26486 webserver.cc:460] Webserver started at http://127.6.209.194:33201/ using document root <none> and password file <none>
I20260926 15:19:45.414547 26486 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:45.414682 26486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:45.414964 26486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:45.417598 26486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "8866e41bbb3c48428d6682857134d857"
format_stamp: "Formatted at 2026-09-26 15:19:45 on dist-test-slave-j6q7"
server_key: "b21b2fa06d1f4bf7f8c92c55cfcf505c"
server_key_iv: "ca7302bbe11e3099f27bfe4ad08263d8"
server_key_version: "encryptionkey@0"
I20260926 15:19:45.418318 26486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "8866e41bbb3c48428d6682857134d857"
format_stamp: "Formatted at 2026-09-26 15:19:45 on dist-test-slave-j6q7"
server_key: "b21b2fa06d1f4bf7f8c92c55cfcf505c"
server_key_iv: "ca7302bbe11e3099f27bfe4ad08263d8"
server_key_version: "encryptionkey@0"
I20260926 15:19:45.423273 26486 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 15:19:45.426478 26501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:45.427645 26486 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 15:19:45.427804 26486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "8866e41bbb3c48428d6682857134d857"
format_stamp: "Formatted at 2026-09-26 15:19:45 on dist-test-slave-j6q7"
server_key: "b21b2fa06d1f4bf7f8c92c55cfcf505c"
server_key_iv: "ca7302bbe11e3099f27bfe4ad08263d8"
server_key_version: "encryptionkey@0"
I20260926 15:19:45.427978 26486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:45.438764 26486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:45.442595 26486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:45.442891 26486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:45.443763 26486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:19:45.445195 26486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:19:45.445307 26486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:45.445461 26486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:19:45.445540 26486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:45.475338 26486 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:38507
I20260926 15:19:45.475375 26613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:38507 every 8 connection(s)
I20260926 15:19:45.476460 26486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:19:45.477896 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 26486
I20260926 15:19:45.478044 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:19:45.478420 6983 external_mini_cluster.cc:1428] Setting key 9831058a473561ddd2e3067fe5e57a76
Sep 26 15:19:45 dist-test-slave-j6q7 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435985, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.194@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:19:45.498121 26482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:45.478786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:44635 (local address 127.6.209.254:42571)
0926 15:19:45.478966 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:45.478975 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:45.480106 (+ 1131us) server_negotiation.cc:407] Connection header received
0926 15:19:45.482756 (+ 2650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:45.482762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:45.482837 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:45.482936 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:45.484739 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:45.485845 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:45.488082 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:45.488695 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:45.488850 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:45.491924 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:45.491932 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:45.491956 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:45.491999 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:45.495317 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:45.495929 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:45.495936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:45.495942 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:45.496083 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:45.496460 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:45.496467 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:45.496472 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:45.496851 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:45.496998 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:45.497526 (+ 528us) server_negotiation.cc:299] Negotiation successful
0926 15:19:45.497748 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 15:19:45.498600 26616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:45.479232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:42571 (local address 127.6.209.194:44635)
0926 15:19:45.479969 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:45.480018 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:45.482479 (+ 2461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:45.483181 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:45.483194 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:45.483894 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:19:45.484566 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:45.484589 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:45.486001 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:45.486008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:45.487934 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:45.487950 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:45.488935 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:45.488943 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:45.489229 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:45.489826 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:19:45.489845 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:19:45.491747 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:19:45.495440 (+ 3693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:19:45.495445 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:19:45.495454 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:19:45.495804 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:19:45.496179 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:19:45.496181 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:19:45.496184 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:19:45.496352 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:19:45.496957 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:19:45.496961 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:19:45.497206 (+ 245us) client_negotiation.cc:769] Sending connection context
0926 15:19:45.497398 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 15:19:45.497727 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":296,"threads_started":1}
I20260926 15:19:45.499840 26614 heartbeater.cc:347] Connected to a master server at 127.6.209.254:42571
I20260926 15:19:45.500156 26614 heartbeater.cc:464] Registering TS with master...
I20260926 15:19:45.500785 26614 heartbeater.cc:511] Master 127.6.209.254:42571 requested a full tablet report, sending...
I20260926 15:19:45.502158 26291 ts_manager.cc:194] Registered new tserver with Master: 8866e41bbb3c48428d6682857134d857 (127.6.209.194:38507)
I20260926 15:19:45.502920 26291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.194@KRBTEST.COM'} at 127.6.209.194:44635
WARNING: no policy specified for kudu/127.6.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
Entry for principal kudu/127.6.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab.
I20260926 15:19:45.528934 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:42571
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:39375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:19:45.822953 26619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:45.823205 26619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:45.823284 26619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:45.829561 26619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:45.829648 26619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:45.829761 26619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:19:45.838294 26619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:39375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.195.keytab
--principal=kudu/127.6.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:42571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:45.839280 26619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:45.840500 26619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:45.852033 26625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:45.852239 26624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:45.851960 26627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:45.852950 26619 server_base.cc:1034] running on GCE node
I20260926 15:19:45.853470 26619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:45.854223 26619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:45.855419 26619 hybrid_clock.cc:648] HybridClock initialized: now 1790435985855383 us; error 47 us; skew 500 ppm
I20260926 15:19:45.855686 26619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:19:45 dist-test-slave-j6q7 krb5kdc[26246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435985, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:19:45.858270 26619 init.cc:377] Logged in from keytab as kudu/127.6.209.195@KRBTEST.COM (short username kudu)
I20260926 15:19:45.859463 26619 webserver.cc:460] Webserver started at http://127.6.209.195:40241/ using document root <none> and password file <none>
I20260926 15:19:45.860069 26619 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:45.860209 26619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:45.860488 26619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:45.863027 26619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "6522614fa21c433daa3b81d51cad95b5"
format_stamp: "Formatted at 2026-09-26 15:19:45 on dist-test-slave-j6q7"
server_key: "2cde8db5778e44ada831e23ce0830ca7"
server_key_iv: "3c3983230cab83927f89f35f64948302"
server_key_version: "encryptionkey@0"
I20260926 15:19:45.863777 26619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "6522614fa21c433daa3b81d51cad95b5"
format_stamp: "Formatted at 2026-09-26 15:19:45 on dist-test-slave-j6q7"
server_key: "2cde8db5778e44ada831e23ce0830ca7"
server_key_iv: "3c3983230cab83927f89f35f64948302"
server_key_version: "encryptionkey@0"
I20260926 15:19:45.867880 26619 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 15:19:45.870677 26634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:45.871825 26619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:19:45.871997 26619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "6522614fa21c433daa3b81d51cad95b5"
format_stamp: "Formatted at 2026-09-26 15:19:45 on dist-test-slave-j6q7"
server_key: "2cde8db5778e44ada831e23ce0830ca7"
server_key_iv: "3c3983230cab83927f89f35f64948302"
server_key_version: "encryptionkey@0"
I20260926 15:19:45.872174 26619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:45.897568 26619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:45.900938 26619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:45.901207 26619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:45.901970 26619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:19:45.903208 26619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:19:45.903311 26619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:45.903440 26619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:19:45.903513 26619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:45.936795 26619 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:36519
I20260926 15:19:45.936887 26746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:36519 every 8 connection(s)
I20260926 15:19:45.937937 26619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:19:45.946580 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 26619
I20260926 15:19:45.946759 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:19:45.947180 6983 external_mini_cluster.cc:1428] Setting key 06f4a79f5da46e87821bc816caa9268d
Sep 26 15:19:45 dist-test-slave-j6q7 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435985, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.195@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:19:45.955248 26482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:45.940061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:46409 (local address 127.6.209.254:42571)
0926 15:19:45.940260 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:45.940267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:45.941535 (+ 1268us) server_negotiation.cc:407] Connection header received
0926 15:19:45.943142 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:45.943146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:45.943230 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:45.943308 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:45.944506 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:45.945226 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:45.946622 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:45.947030 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:45.947112 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:45.950563 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:45.950568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:45.950571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:45.950597 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:45.953163 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:45.953726 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:45.953730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:45.953732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:45.953831 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:45.954141 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:45.954143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:45.954145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:45.954354 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:45.954447 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:45.954917 (+ 470us) server_negotiation.cc:299] Negotiation successful
0926 15:19:45.955048 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 15:19:45.955971 26749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:45.940677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:42571 (local address 127.6.209.195:46409)
0926 15:19:45.941405 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:45.941438 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:45.942916 (+ 1478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:45.943490 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:45.943498 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:45.943901 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:19:45.944364 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:45.944378 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:45.945342 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:45.945345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:45.946505 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:45.946518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:45.947233 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:45.947241 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:45.947479 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:45.948070 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:19:45.948110 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:19:45.950405 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:19:45.953302 (+ 2897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:19:45.953307 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:19:45.953315 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:19:45.953639 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:19:45.953922 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:19:45.953924 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:19:45.953926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:19:45.954062 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:19:45.954442 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:19:45.954446 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:19:45.954672 (+ 226us) client_negotiation.cc:769] Sending connection context
0926 15:19:45.954846 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 15:19:45.955112 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":323,"threads_started":1}
I20260926 15:19:45.957110 26747 heartbeater.cc:347] Connected to a master server at 127.6.209.254:42571
I20260926 15:19:45.957376 26747 heartbeater.cc:464] Registering TS with master...
I20260926 15:19:45.957979 26747 heartbeater.cc:511] Master 127.6.209.254:42571 requested a full tablet report, sending...
I20260926 15:19:45.959153 26291 ts_manager.cc:194] Registered new tserver with Master: 6522614fa21c433daa3b81d51cad95b5 (127.6.209.195:36519)
I20260926 15:19:45.959887 26291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.209.195@KRBTEST.COM'} at 127.6.209.195:46409
I20260926 15:19:45.961707 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:19:45.975709 26482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:45.965425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44882 (local address 127.6.209.254:42571)
0926 15:19:45.965584 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:45.965591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:45.965901 (+ 310us) server_negotiation.cc:407] Connection header received
0926 15:19:45.966007 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:45.966010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:45.966054 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:45.966117 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:45.967195 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:45.967783 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:45.968872 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:45.969201 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:45.969303 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:45.971134 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:45.971138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:45.971140 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:45.971164 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:45.973584 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:45.974119 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:45.974122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:45.974124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:45.974222 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:45.974542 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:45.974544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:45.974546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:45.974817 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:45.975023 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:45.975284 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 15:19:45.975439 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 15:19:45.978842 26291 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:19:45.985275 26291 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:19:46.014865 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:45.995646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39208 (local address 127.6.209.193:32835)
0926 15:19:46.007086 (+ 11440us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:46.007095 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:46.007119 (+ 24us) server_negotiation.cc:407] Connection header received
0926 15:19:46.007248 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:46.007257 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:46.007493 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:46.008336 (+ 843us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:46.008526 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.009885 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.012079 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.013778 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.013917 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:46.014097 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:46.014383 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 15:19:46.014593 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11279,"thread_start_us":11561,"threads_started":1}
I20260926 15:19:46.014959 26759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:45.995470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:32835 (local address 127.0.0.1:39208)
0926 15:19:46.002539 (+ 7069us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:46.002601 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:46.002794 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:46.007733 (+ 4939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:46.007739 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:46.007768 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:46.008114 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:46.008138 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.010021 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.010026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:46.011899 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:46.011913 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.013973 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.013979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:46.014142 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:46.014188 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 15:19:46.014339 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 15:19:46.014521 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6774,"mutex_wait_us":228,"thread_start_us":6622,"threads_started":1}
I20260926 15:19:46.015494 26763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:45.997137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:36519 (local address 127.0.0.1:57626)
0926 15:19:46.007292 (+ 10155us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:46.007312 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:46.007406 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:46.008362 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:46.008365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:46.008386 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:46.008678 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:46.008689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.010743 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.010749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:46.013106 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:46.013120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.014861 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.014865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:46.015030 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:46.015069 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 15:19:46.015132 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 15:19:46.015229 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10023,"mutex_wait_us":180,"thread_start_us":9914,"threads_started":1}
I20260926 15:19:46.015748 26761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:45.996208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55472 (local address 127.6.209.194:38507)
0926 15:19:45.996913 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:45.996922 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:46.007275 (+ 10353us) server_negotiation.cc:407] Connection header received
0926 15:19:46.007360 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:46.007366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:46.007531 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:46.007720 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:46.008463 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.009726 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.012504 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.013918 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.014025 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:46.014164 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:46.015349 (+ 1185us) server_negotiation.cc:299] Negotiation successful
0926 15:19:46.015492 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":1102,"threads_started":1}
I20260926 15:19:46.016220 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:45.996802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:38507 (local address 127.0.0.1:55472)
0926 15:19:46.006636 (+ 9834us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:46.006662 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:46.006805 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:46.007771 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:46.007775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:46.007800 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:46.008270 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:46.008297 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.009841 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.009847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:46.012349 (+ 2502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:46.012363 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.014285 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.014289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:46.014453 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:46.014495 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 15:19:46.014570 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 15:19:46.014668 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9674,"mutex_wait_us":52,"thread_start_us":9512,"threads_started":1}
I20260926 15:19:46.016376 26764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:45.997491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57626 (local address 127.6.209.195:36519)
0926 15:19:46.007811 (+ 10320us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:46.007818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:46.007846 (+ 28us) server_negotiation.cc:407] Connection header received
0926 15:19:46.007932 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:46.007941 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:46.008151 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:46.008368 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:46.008870 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.010593 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.013272 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.014604 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.014714 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:46.015933 (+ 1219us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:46.015989 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 15:19:46.016088 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10190,"thread_start_us":10022,"threads_started":1}
I20260926 15:19:46.018628 26415 tablet_service.cc:1467] Processing CreateTablet for tablet 6656764334884e928f4f89072c573847 (DEFAULT_TABLE table=test-table [id=0cb98dd41e4d499ea70ddd35182ac364]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:19:46.019064 26549 tablet_service.cc:1467] Processing CreateTablet for tablet 6656764334884e928f4f89072c573847 (DEFAULT_TABLE table=test-table [id=0cb98dd41e4d499ea70ddd35182ac364]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:19:46.019443 26682 tablet_service.cc:1467] Processing CreateTablet for tablet 6656764334884e928f4f89072c573847 (DEFAULT_TABLE table=test-table [id=0cb98dd41e4d499ea70ddd35182ac364]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:19:46.020025 26415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6656764334884e928f4f89072c573847. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:46.020201 26549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6656764334884e928f4f89072c573847. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:46.020712 26682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6656764334884e928f4f89072c573847. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:46.033418 26765 tablet_bootstrap.cc:492] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5: Bootstrap starting.
I20260926 15:19:46.037225 26765 tablet_bootstrap.cc:654] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:46.037426 26766 tablet_bootstrap.cc:492] T 6656764334884e928f4f89072c573847 P 6f8a6f4b3d214315a9addd6adeacba8f: Bootstrap starting.
I20260926 15:19:46.038518 26765 log.cc:824] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:46.040110 26766 tablet_bootstrap.cc:654] T 6656764334884e928f4f89072c573847 P 6f8a6f4b3d214315a9addd6adeacba8f: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:46.041215 26766 log.cc:824] T 6656764334884e928f4f89072c573847 P 6f8a6f4b3d214315a9addd6adeacba8f: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:46.043469 26766 tablet_bootstrap.cc:492] T 6656764334884e928f4f89072c573847 P 6f8a6f4b3d214315a9addd6adeacba8f: No bootstrap required, opened a new log
I20260926 15:19:46.043722 26766 ts_tablet_manager.cc:1397] T 6656764334884e928f4f89072c573847 P 6f8a6f4b3d214315a9addd6adeacba8f: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260926 15:19:46.044740 26767 tablet_bootstrap.cc:492] T 6656764334884e928f4f89072c573847 P 8866e41bbb3c48428d6682857134d857: Bootstrap starting.
I20260926 15:19:46.047513 26765 tablet_bootstrap.cc:492] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5: No bootstrap required, opened a new log
I20260926 15:19:46.047729 26765 ts_tablet_manager.cc:1397] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260926 15:19:46.048055 26767 tablet_bootstrap.cc:654] T 6656764334884e928f4f89072c573847 P 8866e41bbb3c48428d6682857134d857: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:46.048879 26767 log.cc:824] T 6656764334884e928f4f89072c573847 P 8866e41bbb3c48428d6682857134d857: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:46.051752 26480 heartbeater.cc:503] Master 127.6.209.254:42571 was elected leader, sending a full tablet report...
I20260926 15:19:46.051829 26767 tablet_bootstrap.cc:492] T 6656764334884e928f4f89072c573847 P 8866e41bbb3c48428d6682857134d857: No bootstrap required, opened a new log
I20260926 15:19:46.052014 26767 ts_tablet_manager.cc:1397] T 6656764334884e928f4f89072c573847 P 8866e41bbb3c48428d6682857134d857: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260926 15:19:46.051949 26766 raft_consensus.cc:348] T 6656764334884e928f4f89072c573847 P 6f8a6f4b3d214315a9addd6adeacba8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8a6f4b3d214315a9addd6adeacba8f" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 32835 } } peers { permanent_uuid: "8866e41bbb3c48428d6682857134d857" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 38507 } } peers { permanent_uuid: "6522614fa21c433daa3b81d51cad95b5" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36519 } }
I20260926 15:19:46.052392 26766 raft_consensus.cc:374] T 6656764334884e928f4f89072c573847 P 6f8a6f4b3d214315a9addd6adeacba8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:46.052510 26766 raft_consensus.cc:729] T 6656764334884e928f4f89072c573847 P 6f8a6f4b3d214315a9addd6adeacba8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f8a6f4b3d214315a9addd6adeacba8f, State: Initialized, Role: FOLLOWER
I20260926 15:19:46.053094 26766 consensus_queue.cc:260] T 6656764334884e928f4f89072c573847 P 6f8a6f4b3d214315a9addd6adeacba8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8a6f4b3d214315a9addd6adeacba8f" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 32835 } } peers { permanent_uuid: "8866e41bbb3c48428d6682857134d857" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 38507 } } peers { permanent_uuid: "6522614fa21c433daa3b81d51cad95b5" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36519 } }
I20260926 15:19:46.058420 26765 raft_consensus.cc:348] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8a6f4b3d214315a9addd6adeacba8f" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 32835 } } peers { permanent_uuid: "8866e41bbb3c48428d6682857134d857" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 38507 } } peers { permanent_uuid: "6522614fa21c433daa3b81d51cad95b5" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36519 } }
I20260926 15:19:46.058938 26765 raft_consensus.cc:374] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:46.059053 26765 raft_consensus.cc:729] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6522614fa21c433daa3b81d51cad95b5, State: Initialized, Role: FOLLOWER
I20260926 15:19:46.059701 26765 consensus_queue.cc:260] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8a6f4b3d214315a9addd6adeacba8f" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 32835 } } peers { permanent_uuid: "8866e41bbb3c48428d6682857134d857" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 38507 } } peers { permanent_uuid: "6522614fa21c433daa3b81d51cad95b5" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36519 } }
I20260926 15:19:46.060585 26767 raft_consensus.cc:348] T 6656764334884e928f4f89072c573847 P 8866e41bbb3c48428d6682857134d857 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8a6f4b3d214315a9addd6adeacba8f" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 32835 } } peers { permanent_uuid: "8866e41bbb3c48428d6682857134d857" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 38507 } } peers { permanent_uuid: "6522614fa21c433daa3b81d51cad95b5" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36519 } }
I20260926 15:19:46.060925 26747 heartbeater.cc:503] Master 127.6.209.254:42571 was elected leader, sending a full tablet report...
I20260926 15:19:46.061096 26767 raft_consensus.cc:374] T 6656764334884e928f4f89072c573847 P 8866e41bbb3c48428d6682857134d857 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:46.061229 26767 raft_consensus.cc:729] T 6656764334884e928f4f89072c573847 P 8866e41bbb3c48428d6682857134d857 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8866e41bbb3c48428d6682857134d857, State: Initialized, Role: FOLLOWER
I20260926 15:19:46.061733 26765 ts_tablet_manager.cc:1428] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260926 15:19:46.061815 26767 consensus_queue.cc:260] T 6656764334884e928f4f89072c573847 P 8866e41bbb3c48428d6682857134d857 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8a6f4b3d214315a9addd6adeacba8f" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 32835 } } peers { permanent_uuid: "8866e41bbb3c48428d6682857134d857" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 38507 } } peers { permanent_uuid: "6522614fa21c433daa3b81d51cad95b5" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36519 } }
I20260926 15:19:46.063009 26614 heartbeater.cc:503] Master 127.6.209.254:42571 was elected leader, sending a full tablet report...
I20260926 15:19:46.063573 26766 ts_tablet_manager.cc:1428] T 6656764334884e928f4f89072c573847 P 6f8a6f4b3d214315a9addd6adeacba8f: Time spent starting tablet: real 0.020s user 0.006s sys 0.013s
I20260926 15:19:46.065497 26767 ts_tablet_manager.cc:1428] T 6656764334884e928f4f89072c573847 P 8866e41bbb3c48428d6682857134d857: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260926 15:19:46.108062 26772 raft_consensus.cc:482] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:19:46.108314 26772 raft_consensus.cc:504] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8a6f4b3d214315a9addd6adeacba8f" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 32835 } } peers { permanent_uuid: "8866e41bbb3c48428d6682857134d857" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 38507 } } peers { permanent_uuid: "6522614fa21c433daa3b81d51cad95b5" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36519 } }
I20260926 15:19:46.109756 26772 leader_election.cc:288] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f8a6f4b3d214315a9addd6adeacba8f (127.6.209.193:32835), 8866e41bbb3c48428d6682857134d857 (127.6.209.194:38507)
I20260926 15:19:46.114657 26761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:46.110241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:56319 (local address 127.6.209.194:38507)
0926 15:19:46.110423 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:46.110429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:46.110450 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:19:46.110494 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:46.110497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:46.110542 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:46.110623 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:46.111039 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.111626 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.113195 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.113913 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.114055 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:46.114115 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:46.114339 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 15:19:46.114443 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 15:19:46.114677 26749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:46.109991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:38507 (local address 127.6.209.195:56319)
0926 15:19:46.110188 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:46.110209 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:46.110317 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:46.110651 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:46.110655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:46.110679 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:46.110911 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:46.110927 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.111744 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.111750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:46.113032 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:46.113043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.114055 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.114060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:46.114206 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:46.114226 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 15:19:46.114317 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 15:19:46.114428 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"spinlock_wait_cycles":5376}
I20260926 15:19:46.115383 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:46.110091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:38471 (local address 127.6.209.193:32835)
0926 15:19:46.110256 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:46.110261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:46.110668 (+ 407us) server_negotiation.cc:407] Connection header received
0926 15:19:46.110843 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:46.110849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:46.110903 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:46.110994 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:46.111440 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.112032 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.113671 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.114437 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.114581 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:46.114642 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:46.115057 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 15:19:46.115167 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 15:19:46.115456 26774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:46.110060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:32835 (local address 127.6.209.195:38471)
0926 15:19:46.110571 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:46.110583 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:46.110719 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:46.111007 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:46.111010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:46.111029 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:46.111295 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:46.111302 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.112202 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.112207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:46.113535 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:46.113545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.114615 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.114627 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:46.114816 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:46.114846 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 15:19:46.115034 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 15:19:46.115136 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":3,"thread_start_us":634,"threads_started":1}
I20260926 15:19:46.115486 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6656764334884e928f4f89072c573847" candidate_uuid: "6522614fa21c433daa3b81d51cad95b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8866e41bbb3c48428d6682857134d857" is_pre_election: true
I20260926 15:19:46.115918 26569 raft_consensus.cc:2454] T 6656764334884e928f4f89072c573847 P 8866e41bbb3c48428d6682857134d857 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6522614fa21c433daa3b81d51cad95b5 in term 0.
I20260926 15:19:46.116362 26435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6656764334884e928f4f89072c573847" candidate_uuid: "6522614fa21c433daa3b81d51cad95b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f8a6f4b3d214315a9addd6adeacba8f" is_pre_election: true
I20260926 15:19:46.116513 26636 leader_election.cc:302] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6522614fa21c433daa3b81d51cad95b5, 8866e41bbb3c48428d6682857134d857; no voters:
I20260926 15:19:46.116849 26772 raft_consensus.cc:2781] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:19:46.116832 26435 raft_consensus.cc:2454] T 6656764334884e928f4f89072c573847 P 6f8a6f4b3d214315a9addd6adeacba8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6522614fa21c433daa3b81d51cad95b5 in term 0.
I20260926 15:19:46.116982 26772 raft_consensus.cc:482] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:19:46.117085 26772 raft_consensus.cc:3037] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:46.119019 26772 raft_consensus.cc:504] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8a6f4b3d214315a9addd6adeacba8f" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 32835 } } peers { permanent_uuid: "8866e41bbb3c48428d6682857134d857" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 38507 } } peers { permanent_uuid: "6522614fa21c433daa3b81d51cad95b5" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36519 } }
I20260926 15:19:46.119490 26772 leader_election.cc:288] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5 [CANDIDATE]: Term 1 election: Requested vote from peers 6f8a6f4b3d214315a9addd6adeacba8f (127.6.209.193:32835), 8866e41bbb3c48428d6682857134d857 (127.6.209.194:38507)
I20260926 15:19:46.119917 26435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6656764334884e928f4f89072c573847" candidate_uuid: "6522614fa21c433daa3b81d51cad95b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f8a6f4b3d214315a9addd6adeacba8f"
I20260926 15:19:46.119967 26569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6656764334884e928f4f89072c573847" candidate_uuid: "6522614fa21c433daa3b81d51cad95b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8866e41bbb3c48428d6682857134d857"
I20260926 15:19:46.120083 26435 raft_consensus.cc:3037] T 6656764334884e928f4f89072c573847 P 6f8a6f4b3d214315a9addd6adeacba8f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:46.120193 26569 raft_consensus.cc:3037] T 6656764334884e928f4f89072c573847 P 8866e41bbb3c48428d6682857134d857 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:46.122253 26435 raft_consensus.cc:2454] T 6656764334884e928f4f89072c573847 P 6f8a6f4b3d214315a9addd6adeacba8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6522614fa21c433daa3b81d51cad95b5 in term 1.
I20260926 15:19:46.122464 26569 raft_consensus.cc:2454] T 6656764334884e928f4f89072c573847 P 8866e41bbb3c48428d6682857134d857 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6522614fa21c433daa3b81d51cad95b5 in term 1.
I20260926 15:19:46.122674 26637 leader_election.cc:302] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6522614fa21c433daa3b81d51cad95b5, 6f8a6f4b3d214315a9addd6adeacba8f; no voters:
I20260926 15:19:46.122989 26772 raft_consensus.cc:2781] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:19:46.123370 26772 raft_consensus.cc:686] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5 [term 1 LEADER]: Becoming Leader. State: Replica: 6522614fa21c433daa3b81d51cad95b5, State: Running, Role: LEADER
I20260926 15:19:46.123790 26772 consensus_queue.cc:237] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8a6f4b3d214315a9addd6adeacba8f" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 32835 } } peers { permanent_uuid: "8866e41bbb3c48428d6682857134d857" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 38507 } } peers { permanent_uuid: "6522614fa21c433daa3b81d51cad95b5" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36519 } }
I20260926 15:19:46.127750 26290 catalog_manager.cc:5495] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6522614fa21c433daa3b81d51cad95b5 (127.6.209.195). New cstate: current_term: 1 leader_uuid: "6522614fa21c433daa3b81d51cad95b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8a6f4b3d214315a9addd6adeacba8f" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 32835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8866e41bbb3c48428d6682857134d857" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6522614fa21c433daa3b81d51cad95b5" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 36519 } health_report { overall_health: HEALTHY } } }
I20260926 15:19:46.166980 26764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:46.162724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57628 (local address 127.6.209.195:36519)
0926 15:19:46.162891 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:46.162895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:46.163004 (+ 109us) server_negotiation.cc:407] Connection header received
0926 15:19:46.163208 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:46.163212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:46.163289 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:46.163400 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:19:46.163695 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.164280 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.165431 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.165775 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.165855 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:46.166210 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:19:46.166601 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:19:46.166702 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:46.166759 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 15:19:46.166819 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 15:19:46.174101 26435 raft_consensus.cc:1261] T 6656764334884e928f4f89072c573847 P 6f8a6f4b3d214315a9addd6adeacba8f [term 1 FOLLOWER]: Refusing update from remote peer 6522614fa21c433daa3b81d51cad95b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:19:46.174101 26569 raft_consensus.cc:1261] T 6656764334884e928f4f89072c573847 P 8866e41bbb3c48428d6682857134d857 [term 1 FOLLOWER]: Refusing update from remote peer 6522614fa21c433daa3b81d51cad95b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:19:46.174988 26775 consensus_queue.cc:1035] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8866e41bbb3c48428d6682857134d857" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:19:46.175280 26772 consensus_queue.cc:1035] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f8a6f4b3d214315a9addd6adeacba8f" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 32835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:19:46.185930 26779 mvcc.cc:205] Tried to move back new op lower bound from 7333625799359504384 to 7333625799166291968. Current Snapshot: MvccSnapshot[applied={T|T < 7333625799359504384}]
I20260926 15:19:46.196683 26290 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:19:46.196942 26290 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:19:46.199743 26290 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c1e7db9f2424ee4928e7675ab209254: Sending DeleteTablet for 3 replicas of tablet 6656764334884e928f4f89072c573847
I20260926 15:19:46.200711 26415 tablet_service.cc:1514] Processing DeleteTablet for tablet 6656764334884e928f4f89072c573847 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:19:46 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:39208
I20260926 15:19:46.200757 26549 tablet_service.cc:1514] Processing DeleteTablet for tablet 6656764334884e928f4f89072c573847 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:19:46 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:55472
I20260926 15:19:46.201001 26682 tablet_service.cc:1514] Processing DeleteTablet for tablet 6656764334884e928f4f89072c573847 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:19:46 UTC) from {username='kudu', principal='kudu/127.6.209.254@KRBTEST.COM'} at 127.0.0.1:57626
I20260926 15:19:46.204566 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 26352
I20260926 15:19:46.213145 26789 tablet_replica.cc:330] T 6656764334884e928f4f89072c573847 P 8866e41bbb3c48428d6682857134d857: stopping tablet replica
I20260926 15:19:46.215184 26789 raft_consensus.cc:2229] T 6656764334884e928f4f89072c573847 P 8866e41bbb3c48428d6682857134d857 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:19:46.217255 26789 raft_consensus.cc:2258] T 6656764334884e928f4f89072c573847 P 8866e41bbb3c48428d6682857134d857 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:19:46.218071 26790 tablet_replica.cc:330] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5: stopping tablet replica
I20260926 15:19:46.218406 26790 raft_consensus.cc:2229] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:19:46.218443 26788 tablet_replica.cc:330] T 6656764334884e928f4f89072c573847 P 6f8a6f4b3d214315a9addd6adeacba8f: stopping tablet replica
I20260926 15:19:46.218721 26788 raft_consensus.cc:2229] T 6656764334884e928f4f89072c573847 P 6f8a6f4b3d214315a9addd6adeacba8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:19:46.218948 26790 raft_consensus.cc:2258] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:19:46.219872 26789 ts_tablet_manager.cc:1905] T 6656764334884e928f4f89072c573847 P 8866e41bbb3c48428d6682857134d857: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:19:46.220366 26788 raft_consensus.cc:2258] T 6656764334884e928f4f89072c573847 P 6f8a6f4b3d214315a9addd6adeacba8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:19:46.220679 26790 ts_tablet_manager.cc:1905] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:19:46.223646 26788 ts_tablet_manager.cc:1905] T 6656764334884e928f4f89072c573847 P 6f8a6f4b3d214315a9addd6adeacba8f: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 15:19:46 dist-test-slave-j6q7 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.193@KRBTEST.COM
I20260926 15:19:46.227470 26790 ts_tablet_manager.cc:1918] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:19:46.227633 26790 log.cc:1196] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/6656764334884e928f4f89072c573847
I20260926 15:19:46.228231 26790 ts_tablet_manager.cc:1939] T 6656764334884e928f4f89072c573847 P 6522614fa21c433daa3b81d51cad95b5: Deleting consensus metadata
I20260926 15:19:46.229875 26278 catalog_manager.cc:4841] TS 6522614fa21c433daa3b81d51cad95b5 (127.6.209.195:36519): tablet 6656764334884e928f4f89072c573847 (table test-table [id=0cb98dd41e4d499ea70ddd35182ac364]) successfully deleted
I20260926 15:19:46.231900 26789 ts_tablet_manager.cc:1918] T 6656764334884e928f4f89072c573847 P 8866e41bbb3c48428d6682857134d857: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:19:46.232089 26789 log.cc:1196] T 6656764334884e928f4f89072c573847 P 8866e41bbb3c48428d6682857134d857: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/6656764334884e928f4f89072c573847
I20260926 15:19:46.232652 26789 ts_tablet_manager.cc:1939] T 6656764334884e928f4f89072c573847 P 8866e41bbb3c48428d6682857134d857: Deleting consensus metadata
I20260926 15:19:46.233713 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:46.206560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39218 (local address 127.6.209.193:32835)
0926 15:19:46.206716 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:46.206723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:46.213708 (+ 6985us) server_negotiation.cc:407] Connection header received
0926 15:19:46.214072 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:46.214079 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:46.214222 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:46.214292 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:46.217697 (+ 3405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.218704 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.220330 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.220931 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.221021 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:46.226209 (+ 5188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:46.226221 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:46.226235 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:46.226293 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:46.229892 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:46.230739 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:46.230748 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:46.230751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:46.230913 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:46.231304 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:46.231309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:46.231312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:46.231958 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:46.232381 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:46.233054 (+ 673us) server_negotiation.cc:299] Negotiation successful
0926 15:19:46.233310 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 15:19:46.234403 26276 catalog_manager.cc:4841] TS 8866e41bbb3c48428d6682857134d857 (127.6.209.194:38507): tablet 6656764334884e928f4f89072c573847 (table test-table [id=0cb98dd41e4d499ea70ddd35182ac364]) successfully deleted
I20260926 15:19:46.234900 26788 ts_tablet_manager.cc:1918] T 6656764334884e928f4f89072c573847 P 6f8a6f4b3d214315a9addd6adeacba8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:19:46.235073 26788 log.cc:1196] T 6656764334884e928f4f89072c573847 P 6f8a6f4b3d214315a9addd6adeacba8f: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435694426955-6983-0/minicluster-data/ts-0/wal/wals/6656764334884e928f4f89072c573847
I20260926 15:19:46.235460 26475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:46.318436 26788 ts_tablet_manager.cc:1939] T 6656764334884e928f4f89072c573847 P 6f8a6f4b3d214315a9addd6adeacba8f: Deleting consensus metadata
I20260926 15:19:46.319061 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 26352
W20260926 15:19:46.336179 26277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.209.193:32835 (error 108)
I20260926 15:19:46.336344 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 26486
I20260926 15:19:46.338088 26763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:46.337617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:32835 (local address 127.0.0.1:39220)
0926 15:19:46.337789 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:46.337927 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 15:19:46.338361 26277 catalog_manager.cc:4584] TS 6f8a6f4b3d214315a9addd6adeacba8f (127.6.209.193:32835): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6656764334884e928f4f89072c573847: Network error: Client connection negotiation failed: client connection to 127.6.209.193:32835: connect: Connection refused (error 111)
Sep 26 15:19:46 dist-test-slave-j6q7 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.194@KRBTEST.COM
I20260926 15:19:46.351897 26761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:46.337238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.6.209.194:38507)
0926 15:19:46.337400 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:46.337406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:46.337507 (+ 101us) server_negotiation.cc:407] Connection header received
0926 15:19:46.337671 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:46.337677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:46.337773 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:46.337904 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:46.339464 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.340344 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.342126 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.342499 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.342588 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:46.347006 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:46.347019 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:46.347033 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:46.347063 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:46.349566 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:46.350132 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:46.350139 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:46.350142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:46.350248 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:46.350572 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:46.350577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:46.350580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:46.350907 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:46.351184 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:46.351427 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 15:19:46.351624 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 15:19:46.352530 26609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:46.392016 26763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:46.391493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:32835 (local address 127.0.0.1:39232)
0926 15:19:46.391702 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:46.391839 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 15:19:46.393172 26763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:46.392771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:32835 (local address 127.0.0.1:39238)
0926 15:19:46.392948 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:46.393046 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 15:19:46.433303 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 26486
I20260926 15:19:46.449156 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 26619
I20260926 15:19:46.454186 26763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:46.453710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:32835 (local address 127.0.0.1:39254)
0926 15:19:46.453912 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:46.454000 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 15:19:46.455334 26763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:46.454904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:32835 (local address 127.0.0.1:39266)
0926 15:19:46.455120 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:46.455202 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
Sep 26 15:19:46 dist-test-slave-j6q7 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.195@KRBTEST.COM
I20260926 15:19:46.464442 26764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:46.449853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57632 (local address 127.6.209.195:36519)
0926 15:19:46.450039 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:46.450045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:46.450066 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:19:46.450167 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:46.450172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:46.450290 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:46.450398 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:46.451455 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.452710 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.454147 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:46.454780 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:46.454920 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:46.458741 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:46.458751 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:46.458764 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:46.458791 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:46.461734 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:46.462376 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:46.462392 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:46.462394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:46.462558 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:46.462971 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:46.462978 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:46.462982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:46.463432 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:46.463734 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:46.463892 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 15:19:46.464085 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 15:19:46.465224 26742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:46.535743 26763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:46.535295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:32835 (local address 127.0.0.1:39270)
0926 15:19:46.535506 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:46.535588 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 15:19:46.536716 26763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:46.536391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:32835 (local address 127.0.0.1:39274)
0926 15:19:46.536562 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:46.536625 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 15:19:46.578949 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 26619
I20260926 15:19:46.597234 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 26259
I20260926 15:19:46.598228 26321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:46.703383 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 26259
2026-09-26T15:19:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (3825 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T15:19:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:19:46Z Disabled control of system clock
I20260926 15:19:46.769981 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:35167
--webserver_interface=127.6.209.254
--webserver_port=0
--builtin_ntp_servers=127.6.209.212:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:35167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 15:19:47.076666 26800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:47.076921 26800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:47.077061 26800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:47.083802 26800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:19:47.083914 26800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:47.084000 26800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:47.084098 26800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:19:47.084175 26800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:19:47.092667 26800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:35167
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:35167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:47.093823 26800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:47.095378 26800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:47.106421 26808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:47.106386 26806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:47.106473 26805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:47.107187 26800 server_base.cc:1034] running on GCE node
I20260926 15:19:47.107916 26800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:47.109136 26800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:47.110351 26800 hybrid_clock.cc:648] HybridClock initialized: now 1790435987110325 us; error 34 us; skew 500 ppm
I20260926 15:19:47.110656 26800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:19:47.112696 26800 webserver.cc:460] Webserver started at http://127.6.209.254:43451/ using document root <none> and password file <none>
I20260926 15:19:47.113354 26800 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:47.113484 26800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:47.113786 26800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:47.116411 26800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "4b7d0714d0fc4ca0b4cb21b236aa85e2"
format_stamp: "Formatted at 2026-09-26 15:19:47 on dist-test-slave-j6q7"
server_key: "3c6e2b8026870a1060f61588d99d1203"
server_key_iv: "0a9afb228e745bd6fe4cb2cf86c8a254"
server_key_version: "encryptionkey@0"
I20260926 15:19:47.117225 26800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "4b7d0714d0fc4ca0b4cb21b236aa85e2"
format_stamp: "Formatted at 2026-09-26 15:19:47 on dist-test-slave-j6q7"
server_key: "3c6e2b8026870a1060f61588d99d1203"
server_key_iv: "0a9afb228e745bd6fe4cb2cf86c8a254"
server_key_version: "encryptionkey@0"
I20260926 15:19:47.121830 26800 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 15:19:47.124894 26814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:47.126066 26800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:19:47.126230 26800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "4b7d0714d0fc4ca0b4cb21b236aa85e2"
format_stamp: "Formatted at 2026-09-26 15:19:47 on dist-test-slave-j6q7"
server_key: "3c6e2b8026870a1060f61588d99d1203"
server_key_iv: "0a9afb228e745bd6fe4cb2cf86c8a254"
server_key_version: "encryptionkey@0"
I20260926 15:19:47.126430 26800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:47.149276 26800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:47.149914 26800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:47.150156 26800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:47.165832 26800 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:35167
I20260926 15:19:47.165859 26865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:35167 every 8 connection(s)
I20260926 15:19:47.166872 26800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:19:47.167783 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 26800
I20260926 15:19:47.167924 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:19:47.168414 6983 external_mini_cluster.cc:1428] Setting key 164401aa0cad203a4adc3fa2f3b73829
I20260926 15:19:47.170411 26866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:47.180480 26869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:47.170212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33538 (local address 127.6.209.254:35167)
0926 15:19:47.171525 (+ 1313us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:47.171534 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:47.171568 (+ 34us) server_negotiation.cc:407] Connection header received
0926 15:19:47.173119 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:47.173138 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:47.173558 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:47.173853 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:47.174299 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:47.175369 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:47.176950 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:47.177481 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:47.177586 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:47.178578 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:47.178584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:47.178594 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:47.179039 (+ 445us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:47.179097 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:47.179123 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:47.179199 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:47.179411 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 15:19:47.179703 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":970,"thread_start_us":742,"threads_started":1}
I20260926 15:19:47.181793 26866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7d0714d0fc4ca0b4cb21b236aa85e2: Bootstrap starting.
I20260926 15:19:47.184602 26866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b7d0714d0fc4ca0b4cb21b236aa85e2: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:47.185518 26866 log.cc:824] T 00000000000000000000000000000000 P 4b7d0714d0fc4ca0b4cb21b236aa85e2: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:47.187485 26866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7d0714d0fc4ca0b4cb21b236aa85e2: No bootstrap required, opened a new log
I20260926 15:19:47.193557 26866 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b7d0714d0fc4ca0b4cb21b236aa85e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7d0714d0fc4ca0b4cb21b236aa85e2" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35167 } }
I20260926 15:19:47.193817 26866 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b7d0714d0fc4ca0b4cb21b236aa85e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:47.193879 26866 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b7d0714d0fc4ca0b4cb21b236aa85e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7d0714d0fc4ca0b4cb21b236aa85e2, State: Initialized, Role: FOLLOWER
I20260926 15:19:47.194257 26866 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b7d0714d0fc4ca0b4cb21b236aa85e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7d0714d0fc4ca0b4cb21b236aa85e2" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35167 } }
I20260926 15:19:47.194412 26866 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b7d0714d0fc4ca0b4cb21b236aa85e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:19:47.194514 26866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b7d0714d0fc4ca0b4cb21b236aa85e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:19:47.194621 26866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b7d0714d0fc4ca0b4cb21b236aa85e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:47.196257 26866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b7d0714d0fc4ca0b4cb21b236aa85e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7d0714d0fc4ca0b4cb21b236aa85e2" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35167 } }
I20260926 15:19:47.196689 26866 leader_election.cc:302] T 00000000000000000000000000000000 P 4b7d0714d0fc4ca0b4cb21b236aa85e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b7d0714d0fc4ca0b4cb21b236aa85e2; no voters:
I20260926 15:19:47.197187 26866 leader_election.cc:288] T 00000000000000000000000000000000 P 4b7d0714d0fc4ca0b4cb21b236aa85e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:19:47.197332 26871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b7d0714d0fc4ca0b4cb21b236aa85e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:19:47.197940 26871 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b7d0714d0fc4ca0b4cb21b236aa85e2 [term 1 LEADER]: Becoming Leader. State: Replica: 4b7d0714d0fc4ca0b4cb21b236aa85e2, State: Running, Role: LEADER
I20260926 15:19:47.198375 26871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b7d0714d0fc4ca0b4cb21b236aa85e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7d0714d0fc4ca0b4cb21b236aa85e2" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35167 } }
I20260926 15:19:47.198890 26866 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b7d0714d0fc4ca0b4cb21b236aa85e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:19:47.201076 26873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7d0714d0fc4ca0b4cb21b236aa85e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b7d0714d0fc4ca0b4cb21b236aa85e2. Latest consensus state: current_term: 1 leader_uuid: "4b7d0714d0fc4ca0b4cb21b236aa85e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7d0714d0fc4ca0b4cb21b236aa85e2" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35167 } } }
I20260926 15:19:47.201473 26873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7d0714d0fc4ca0b4cb21b236aa85e2 [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:47.206063 26872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7d0714d0fc4ca0b4cb21b236aa85e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b7d0714d0fc4ca0b4cb21b236aa85e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7d0714d0fc4ca0b4cb21b236aa85e2" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35167 } } }
I20260926 15:19:47.206450 26872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7d0714d0fc4ca0b4cb21b236aa85e2 [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:47.207260 26880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:19:47.211798 26880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:19:47.219816 26880 catalog_manager.cc:1347] Generated new cluster ID: 825c421c7d5942f4bd90a39395aad03a
I20260926 15:19:47.219928 26880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:19:47.248493 26880 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:19:47.249677 26880 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:19:47.257999 26880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b7d0714d0fc4ca0b4cb21b236aa85e2: Generated new TSK 0
I20260926 15:19:47.258805 26880 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:19:47.271008 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.193:0
--local_ip_for_outbound_sockets=127.6.209.193
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35167
--builtin_ntp_servers=127.6.209.212:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 15:19:47.568449 26890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:47.568691 26890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:47.568768 26890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:47.575299 26890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:47.575372 26890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:47.575498 26890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.193
I20260926 15:19:47.583498 26890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.209.193
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.193
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:47.584591 26890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:47.586203 26890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:47.598176 26896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:47.598282 26895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:47.598325 26898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:47.599047 26890 server_base.cc:1034] running on GCE node
I20260926 15:19:47.599603 26890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:47.600545 26890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:47.601749 26890 hybrid_clock.cc:648] HybridClock initialized: now 1790435987601706 us; error 56 us; skew 500 ppm
I20260926 15:19:47.602058 26890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:19:47.604056 26890 webserver.cc:460] Webserver started at http://127.6.209.193:46227/ using document root <none> and password file <none>
I20260926 15:19:47.604602 26890 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:47.604674 26890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:47.604974 26890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:47.607388 26890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-0/data/instance:
uuid: "6df90bb4f0524b3a874e8ac362b8c9f0"
format_stamp: "Formatted at 2026-09-26 15:19:47 on dist-test-slave-j6q7"
server_key: "59ed37f49ddec27796da90f2dffef330"
server_key_iv: "77d81934a69eee885775d7d8aabe0574"
server_key_version: "encryptionkey@0"
I20260926 15:19:47.608002 26890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance:
uuid: "6df90bb4f0524b3a874e8ac362b8c9f0"
format_stamp: "Formatted at 2026-09-26 15:19:47 on dist-test-slave-j6q7"
server_key: "59ed37f49ddec27796da90f2dffef330"
server_key_iv: "77d81934a69eee885775d7d8aabe0574"
server_key_version: "encryptionkey@0"
I20260926 15:19:47.612282 26890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260926 15:19:47.615454 26904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:47.616550 26890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 15:19:47.616658 26890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-0/wal
uuid: "6df90bb4f0524b3a874e8ac362b8c9f0"
format_stamp: "Formatted at 2026-09-26 15:19:47 on dist-test-slave-j6q7"
server_key: "59ed37f49ddec27796da90f2dffef330"
server_key_iv: "77d81934a69eee885775d7d8aabe0574"
server_key_version: "encryptionkey@0"
I20260926 15:19:47.616850 26890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:47.644505 26890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:47.645228 26890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:47.645486 26890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:47.646236 26890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:19:47.647421 26890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:19:47.647501 26890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:47.647568 26890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:19:47.647625 26890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:47.678224 26890 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.193:44473
I20260926 15:19:47.678313 27016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.193:44473 every 8 connection(s)
I20260926 15:19:47.679348 26890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-0/data/info.pb
I20260926 15:19:47.681135 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 26890
I20260926 15:19:47.681281 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-0/wal/instance
I20260926 15:19:47.681771 6983 external_mini_cluster.cc:1428] Setting key 73c71ddeb7f4e85dbcf0bad8f5d4d91a
I20260926 15:19:47.684811 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.194:0
--local_ip_for_outbound_sockets=127.6.209.194
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35167
--builtin_ntp_servers=127.6.209.212:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 15:19:47.692287 27019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:47.682550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:39965 (local address 127.6.209.254:35167)
0926 15:19:47.683031 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:47.683036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:47.684489 (+ 1453us) server_negotiation.cc:407] Connection header received
0926 15:19:47.686214 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:47.686221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:47.686301 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:47.686408 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:47.687408 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:47.688390 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:47.689745 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:47.690368 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:47.690488 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:47.691564 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:47.691571 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:47.691578 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:47.691599 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:47.691645 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:47.691661 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:47.691749 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:47.691965 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 15:19:47.692029 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":263,"threads_started":1}
I20260926 15:19:47.693639 27020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:47.682562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:35167 (local address 127.6.209.193:39965)
0926 15:19:47.684344 (+ 1782us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:47.684384 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:47.685975 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:47.686644 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:47.686669 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:47.686748 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:19:47.687245 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:47.687259 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:47.688526 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:47.688529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:47.689622 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:47.689632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:47.690533 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:47.690537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:47.690703 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:47.691301 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:19:47.691304 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:19:47.691336 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:19:47.691396 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:19:47.691767 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:19:47.691777 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:19:47.691786 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 15:19:47.691952 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 15:19:47.692332 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1299,"thread_start_us":1113,"threads_started":1}
I20260926 15:19:47.695101 27017 heartbeater.cc:347] Connected to a master server at 127.6.209.254:35167
I20260926 15:19:47.695358 27017 heartbeater.cc:464] Registering TS with master...
I20260926 15:19:47.695993 27017 heartbeater.cc:511] Master 127.6.209.254:35167 requested a full tablet report, sending...
I20260926 15:19:47.697811 26831 ts_manager.cc:194] Registered new tserver with Master: 6df90bb4f0524b3a874e8ac362b8c9f0 (127.6.209.193:44473)
I20260926 15:19:47.699160 26831 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.209.193:39965
W20260926 15:19:47.978134 27021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:47.978349 27021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:47.978435 27021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:47.984858 27021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:47.984938 27021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:47.985105 27021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.194
I20260926 15:19:47.992547 27021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.209.194
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.194
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:47.993623 27021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:47.995174 27021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:48.007148 27029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:48.007193 27027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:48.007257 27026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:48.008033 27021 server_base.cc:1034] running on GCE node
I20260926 15:19:48.008534 27021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:48.009378 27021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:48.010604 27021 hybrid_clock.cc:648] HybridClock initialized: now 1790435988010570 us; error 45 us; skew 500 ppm
I20260926 15:19:48.010886 27021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:19:48.013024 27021 webserver.cc:460] Webserver started at http://127.6.209.194:43277/ using document root <none> and password file <none>
I20260926 15:19:48.013661 27021 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:48.013782 27021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:48.014060 27021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:48.016683 27021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-1/data/instance:
uuid: "93565a7b2f784d6bb32de5e92acba59b"
format_stamp: "Formatted at 2026-09-26 15:19:48 on dist-test-slave-j6q7"
server_key: "a4701a40e2243efc0e492c28ac2cd5ae"
server_key_iv: "29538d868d0d741da51a30cf50143db0"
server_key_version: "encryptionkey@0"
I20260926 15:19:48.017412 27021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance:
uuid: "93565a7b2f784d6bb32de5e92acba59b"
format_stamp: "Formatted at 2026-09-26 15:19:48 on dist-test-slave-j6q7"
server_key: "a4701a40e2243efc0e492c28ac2cd5ae"
server_key_iv: "29538d868d0d741da51a30cf50143db0"
server_key_version: "encryptionkey@0"
I20260926 15:19:48.021833 27021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 15:19:48.024928 27035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:48.026103 27021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:19:48.026252 27021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-1/wal
uuid: "93565a7b2f784d6bb32de5e92acba59b"
format_stamp: "Formatted at 2026-09-26 15:19:48 on dist-test-slave-j6q7"
server_key: "a4701a40e2243efc0e492c28ac2cd5ae"
server_key_iv: "29538d868d0d741da51a30cf50143db0"
server_key_version: "encryptionkey@0"
I20260926 15:19:48.026446 27021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:48.054587 27021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:48.055182 27021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:48.055430 27021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:48.056180 27021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:19:48.057469 27021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:19:48.057580 27021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:48.057700 27021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:19:48.057783 27021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:48.086450 27021 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.194:43203
I20260926 15:19:48.086517 27147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.194:43203 every 8 connection(s)
I20260926 15:19:48.087491 27021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-1/data/info.pb
I20260926 15:19:48.088166 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 27021
I20260926 15:19:48.088366 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-1/wal/instance
I20260926 15:19:48.088939 6983 external_mini_cluster.cc:1428] Setting key 8e5a306ac80e14d6246306028606ff84
I20260926 15:19:48.091816 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.209.195:0
--local_ip_for_outbound_sockets=127.6.209.195
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35167
--builtin_ntp_servers=127.6.209.212:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 15:19:48.098805 27019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:48.089950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.194:44923 (local address 127.6.209.254:35167)
0926 15:19:48.090122 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:48.090126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:48.091515 (+ 1389us) server_negotiation.cc:407] Connection header received
0926 15:19:48.093324 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:48.093329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:48.093425 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:48.093535 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:48.094419 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.095048 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.096372 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.096715 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.096852 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:48.098096 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:48.098101 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:48.098104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:48.098118 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:48.098148 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:48.098159 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:48.098223 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:48.098464 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 15:19:48.098532 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 15:19:48.100070 27150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:48.090565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:35167 (local address 127.6.209.194:44923)
0926 15:19:48.091368 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:48.091397 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:48.093078 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:48.093727 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:48.093735 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:48.093809 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:19:48.094279 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:48.094293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.095187 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.095193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:48.096239 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:48.096252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.096934 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.096941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:48.097126 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:48.097755 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:19:48.097757 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:19:48.097793 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:19:48.097905 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:19:48.098247 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:19:48.098257 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:19:48.098273 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 15:19:48.098452 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 15:19:48.098841 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":269,"threads_started":1}
I20260926 15:19:48.101387 27148 heartbeater.cc:347] Connected to a master server at 127.6.209.254:35167
I20260926 15:19:48.101622 27148 heartbeater.cc:464] Registering TS with master...
I20260926 15:19:48.102290 27148 heartbeater.cc:511] Master 127.6.209.254:35167 requested a full tablet report, sending...
I20260926 15:19:48.103667 26831 ts_manager.cc:194] Registered new tserver with Master: 93565a7b2f784d6bb32de5e92acba59b (127.6.209.194:43203)
I20260926 15:19:48.104346 26831 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.209.194:44923
W20260926 15:19:48.387877 27151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:48.388170 27151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:48.388324 27151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:48.395268 27151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:48.395395 27151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:48.395561 27151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.209.195
I20260926 15:19:48.404054 27151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.209.195
--webserver_port=0
--tserver_master_addrs=127.6.209.254:35167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.209.195
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:48.405189 27151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:48.406747 27151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:48.418589 27156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:48.418555 27157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:48.419219 27159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:48.419534 27151 server_base.cc:1034] running on GCE node
I20260926 15:19:48.420229 27151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:48.421125 27151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:48.422382 27151 hybrid_clock.cc:648] HybridClock initialized: now 1790435988422353 us; error 49 us; skew 500 ppm
I20260926 15:19:48.422698 27151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:19:48.424865 27151 webserver.cc:460] Webserver started at http://127.6.209.195:44721/ using document root <none> and password file <none>
I20260926 15:19:48.425487 27151 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:48.425617 27151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:48.425956 27151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:48.428529 27151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-2/data/instance:
uuid: "eee2a1bf2d36431099c251ec52579111"
format_stamp: "Formatted at 2026-09-26 15:19:48 on dist-test-slave-j6q7"
server_key: "bab0c71023340e56fde32849b5a050a2"
server_key_iv: "a8e45116a7c19e935725ece28e620ec7"
server_key_version: "encryptionkey@0"
I20260926 15:19:48.429348 27151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance:
uuid: "eee2a1bf2d36431099c251ec52579111"
format_stamp: "Formatted at 2026-09-26 15:19:48 on dist-test-slave-j6q7"
server_key: "bab0c71023340e56fde32849b5a050a2"
server_key_iv: "a8e45116a7c19e935725ece28e620ec7"
server_key_version: "encryptionkey@0"
I20260926 15:19:48.434268 27151 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 15:19:48.437525 27165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:48.438577 27151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:19:48.438758 27151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-2/wal
uuid: "eee2a1bf2d36431099c251ec52579111"
format_stamp: "Formatted at 2026-09-26 15:19:48 on dist-test-slave-j6q7"
server_key: "bab0c71023340e56fde32849b5a050a2"
server_key_iv: "a8e45116a7c19e935725ece28e620ec7"
server_key_version: "encryptionkey@0"
I20260926 15:19:48.438959 27151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:48.458514 27151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:48.459154 27151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:48.459429 27151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:48.460256 27151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:19:48.461549 27151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:19:48.461659 27151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:48.461771 27151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:19:48.461885 27151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:48.491183 27151 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.195:39115
I20260926 15:19:48.491250 27277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.195:39115 every 8 connection(s)
I20260926 15:19:48.492254 27151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-2/data/info.pb
I20260926 15:19:48.494046 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 27151
I20260926 15:19:48.494194 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-2/wal/instance
I20260926 15:19:48.494663 6983 external_mini_cluster.cc:1428] Setting key 909aed3a091e247cd7c902639f8a7a88
I20260926 15:19:48.504302 27019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:48.494871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.195:56827 (local address 127.6.209.254:35167)
0926 15:19:48.495083 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:48.495089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:48.496497 (+ 1408us) server_negotiation.cc:407] Connection header received
0926 15:19:48.498418 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:48.498423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:48.498505 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:48.498607 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:48.499518 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.500230 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.501765 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.502199 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.502356 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:48.503592 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:48.503597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:48.503606 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:48.503621 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:48.503648 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:48.503658 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:48.503740 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:48.503927 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 15:19:48.504008 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 15:19:48.505127 27280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:48.495421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:35167 (local address 127.6.209.195:56827)
0926 15:19:48.496359 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:48.496397 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:48.498163 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:48.498810 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:48.498817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:48.498896 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:19:48.499363 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:48.499376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.500406 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.500410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:48.501643 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:48.501653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.502430 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.502437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:48.502647 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:48.503270 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:19:48.503273 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:19:48.503319 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:19:48.503418 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:19:48.503738 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:19:48.503748 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:19:48.503756 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 15:19:48.503916 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 15:19:48.504212 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":317,"threads_started":1}
I20260926 15:19:48.506212 27278 heartbeater.cc:347] Connected to a master server at 127.6.209.254:35167
I20260926 15:19:48.506470 27278 heartbeater.cc:464] Registering TS with master...
I20260926 15:19:48.507076 27278 heartbeater.cc:511] Master 127.6.209.254:35167 requested a full tablet report, sending...
I20260926 15:19:48.508316 26831 ts_manager.cc:194] Registered new tserver with Master: eee2a1bf2d36431099c251ec52579111 (127.6.209.195:39115)
I20260926 15:19:48.509351 26831 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.209.195:56827
I20260926 15:19:48.509397 6983 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:19:48.518107 27019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:48.513284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33544 (local address 127.6.209.254:35167)
0926 15:19:48.513497 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:48.513505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:48.513859 (+ 354us) server_negotiation.cc:407] Connection header received
0926 15:19:48.513962 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:48.513965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:48.514014 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:48.514080 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:48.514463 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.515059 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.516235 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.516582 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.516660 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:48.517559 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:48.517563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:48.517566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:48.517579 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:48.517602 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:48.517612 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:48.517672 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:48.517778 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 15:19:48.517833 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:19:48.521342 26831 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:19:48.527697 26831 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:19:48.559906 27292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:48.537886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:44473 (local address 127.0.0.1:58108)
0926 15:19:48.549366 (+ 11480us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:48.549406 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:48.549589 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:48.550981 (+ 1392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:48.550989 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:48.551025 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:48.551367 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:48.551386 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.554152 (+ 2766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.554157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:48.557161 (+ 3004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:48.557176 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.559080 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.559085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:48.559264 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:48.559330 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 15:19:48.559454 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 15:19:48.559612 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11223,"thread_start_us":18661,"threads_started":1}
I20260926 15:19:48.560916 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:48.538016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58108 (local address 127.6.209.193:44473)
0926 15:19:48.550369 (+ 12353us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:48.550380 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:48.550412 (+ 32us) server_negotiation.cc:407] Connection header received
0926 15:19:48.550517 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:48.550527 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:48.550780 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:48.550976 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:48.552158 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.553998 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.557400 (+ 3402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.558649 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.558734 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:48.558932 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:48.560408 (+ 1476us) server_negotiation.cc:299] Negotiation successful
0926 15:19:48.560570 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12160,"thread_start_us":13173,"threads_started":1}
I20260926 15:19:48.561795 27290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:48.538041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:43203 (local address 127.0.0.1:32830)
0926 15:19:48.551797 (+ 13756us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:48.551818 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:48.551947 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:48.554585 (+ 2638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:48.554590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:48.554614 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:48.555128 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:48.555137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.557024 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.557030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:48.558890 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:48.558902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.560894 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.560903 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:48.561262 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:48.561291 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 15:19:48.561347 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 15:19:48.561448 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13585,"mutex_wait_us":50,"thread_start_us":11725,"threads_started":1}
I20260926 15:19:48.562978 27293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:48.538671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32830 (local address 127.6.209.194:43203)
0926 15:19:48.553847 (+ 15176us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:48.553854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:48.553878 (+ 24us) server_negotiation.cc:407] Connection header received
0926 15:19:48.553965 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:48.553970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:48.554184 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:48.554755 (+ 571us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:48.555279 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.556865 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.559088 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.560653 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.562148 (+ 1495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:48.562423 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:48.562492 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 15:19:48.562639 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15017,"thread_start_us":15652,"threads_started":1}
I20260926 15:19:48.565326 26952 tablet_service.cc:1467] Processing CreateTablet for tablet 6b990107062e43149cf9e75a7caf1331 (DEFAULT_TABLE table=test-table [id=b995ace9fbdb4d66a1d07f19fdff7a7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:19:48.566720 26952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b990107062e43149cf9e75a7caf1331. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:48.566939 27083 tablet_service.cc:1467] Processing CreateTablet for tablet 6b990107062e43149cf9e75a7caf1331 (DEFAULT_TABLE table=test-table [id=b995ace9fbdb4d66a1d07f19fdff7a7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:19:48.567838 27292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:48.550663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:39115 (local address 127.0.0.1:51388)
0926 15:19:48.560140 (+ 9477us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:48.560156 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:48.560246 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:48.563038 (+ 2792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:48.563044 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:48.563077 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:48.563224 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:48.563236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.564866 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.564871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:48.566021 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:48.566031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.567249 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.567254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:48.567416 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:48.567453 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 15:19:48.567579 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 15:19:48.567671 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9390,"thread_start_us":12960,"threads_started":1}
I20260926 15:19:48.568156 27083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b990107062e43149cf9e75a7caf1331. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:48.568414 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:48.550741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51388 (local address 127.6.209.195:39115)
0926 15:19:48.562481 (+ 11740us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:48.562493 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:48.562530 (+ 37us) server_negotiation.cc:407] Connection header received
0926 15:19:48.562628 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:48.562636 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:48.562848 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:48.563030 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:48.563433 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.564705 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.566183 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.567084 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.567214 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:48.567465 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:48.568035 (+ 570us) server_negotiation.cc:299] Negotiation successful
0926 15:19:48.568138 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11486,"thread_start_us":12500,"threads_started":1}
I20260926 15:19:48.571215 27213 tablet_service.cc:1467] Processing CreateTablet for tablet 6b990107062e43149cf9e75a7caf1331 (DEFAULT_TABLE table=test-table [id=b995ace9fbdb4d66a1d07f19fdff7a7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:19:48.572232 27213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b990107062e43149cf9e75a7caf1331. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:48.579766 27296 tablet_bootstrap.cc:492] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0: Bootstrap starting.
I20260926 15:19:48.581678 27297 tablet_bootstrap.cc:492] T 6b990107062e43149cf9e75a7caf1331 P 93565a7b2f784d6bb32de5e92acba59b: Bootstrap starting.
I20260926 15:19:48.582937 27298 tablet_bootstrap.cc:492] T 6b990107062e43149cf9e75a7caf1331 P eee2a1bf2d36431099c251ec52579111: Bootstrap starting.
I20260926 15:19:48.583880 27296 tablet_bootstrap.cc:654] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:48.584771 27296 log.cc:824] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:48.585610 27297 tablet_bootstrap.cc:654] T 6b990107062e43149cf9e75a7caf1331 P 93565a7b2f784d6bb32de5e92acba59b: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:48.586076 27298 tablet_bootstrap.cc:654] T 6b990107062e43149cf9e75a7caf1331 P eee2a1bf2d36431099c251ec52579111: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:48.586656 27297 log.cc:824] T 6b990107062e43149cf9e75a7caf1331 P 93565a7b2f784d6bb32de5e92acba59b: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:48.587092 27296 tablet_bootstrap.cc:492] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0: No bootstrap required, opened a new log
I20260926 15:19:48.587069 27298 log.cc:824] T 6b990107062e43149cf9e75a7caf1331 P eee2a1bf2d36431099c251ec52579111: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:48.587330 27296 ts_tablet_manager.cc:1397] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 15:19:48.589202 27297 tablet_bootstrap.cc:492] T 6b990107062e43149cf9e75a7caf1331 P 93565a7b2f784d6bb32de5e92acba59b: No bootstrap required, opened a new log
I20260926 15:19:48.589468 27297 ts_tablet_manager.cc:1397] T 6b990107062e43149cf9e75a7caf1331 P 93565a7b2f784d6bb32de5e92acba59b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260926 15:19:48.589767 27298 tablet_bootstrap.cc:492] T 6b990107062e43149cf9e75a7caf1331 P eee2a1bf2d36431099c251ec52579111: No bootstrap required, opened a new log
I20260926 15:19:48.589983 27298 ts_tablet_manager.cc:1397] T 6b990107062e43149cf9e75a7caf1331 P eee2a1bf2d36431099c251ec52579111: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 15:19:48.595978 27297 raft_consensus.cc:348] T 6b990107062e43149cf9e75a7caf1331 P 93565a7b2f784d6bb32de5e92acba59b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df90bb4f0524b3a874e8ac362b8c9f0" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 44473 } } peers { permanent_uuid: "93565a7b2f784d6bb32de5e92acba59b" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 43203 } } peers { permanent_uuid: "eee2a1bf2d36431099c251ec52579111" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39115 } }
I20260926 15:19:48.596294 27297 raft_consensus.cc:374] T 6b990107062e43149cf9e75a7caf1331 P 93565a7b2f784d6bb32de5e92acba59b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:48.596411 27297 raft_consensus.cc:729] T 6b990107062e43149cf9e75a7caf1331 P 93565a7b2f784d6bb32de5e92acba59b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93565a7b2f784d6bb32de5e92acba59b, State: Initialized, Role: FOLLOWER
I20260926 15:19:48.596859 27296 raft_consensus.cc:348] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df90bb4f0524b3a874e8ac362b8c9f0" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 44473 } } peers { permanent_uuid: "93565a7b2f784d6bb32de5e92acba59b" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 43203 } } peers { permanent_uuid: "eee2a1bf2d36431099c251ec52579111" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39115 } }
I20260926 15:19:48.596997 27297 consensus_queue.cc:260] T 6b990107062e43149cf9e75a7caf1331 P 93565a7b2f784d6bb32de5e92acba59b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df90bb4f0524b3a874e8ac362b8c9f0" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 44473 } } peers { permanent_uuid: "93565a7b2f784d6bb32de5e92acba59b" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 43203 } } peers { permanent_uuid: "eee2a1bf2d36431099c251ec52579111" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39115 } }
I20260926 15:19:48.597200 27296 raft_consensus.cc:374] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:48.597278 27296 raft_consensus.cc:729] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6df90bb4f0524b3a874e8ac362b8c9f0, State: Initialized, Role: FOLLOWER
I20260926 15:19:48.597810 27296 consensus_queue.cc:260] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df90bb4f0524b3a874e8ac362b8c9f0" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 44473 } } peers { permanent_uuid: "93565a7b2f784d6bb32de5e92acba59b" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 43203 } } peers { permanent_uuid: "eee2a1bf2d36431099c251ec52579111" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39115 } }
I20260926 15:19:48.598968 27297 ts_tablet_manager.cc:1428] T 6b990107062e43149cf9e75a7caf1331 P 93565a7b2f784d6bb32de5e92acba59b: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260926 15:19:48.599367 27148 heartbeater.cc:503] Master 127.6.209.254:35167 was elected leader, sending a full tablet report...
I20260926 15:19:48.599741 27298 raft_consensus.cc:348] T 6b990107062e43149cf9e75a7caf1331 P eee2a1bf2d36431099c251ec52579111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df90bb4f0524b3a874e8ac362b8c9f0" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 44473 } } peers { permanent_uuid: "93565a7b2f784d6bb32de5e92acba59b" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 43203 } } peers { permanent_uuid: "eee2a1bf2d36431099c251ec52579111" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39115 } }
I20260926 15:19:48.600220 27298 raft_consensus.cc:374] T 6b990107062e43149cf9e75a7caf1331 P eee2a1bf2d36431099c251ec52579111 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:48.600340 27298 raft_consensus.cc:729] T 6b990107062e43149cf9e75a7caf1331 P eee2a1bf2d36431099c251ec52579111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee2a1bf2d36431099c251ec52579111, State: Initialized, Role: FOLLOWER
I20260926 15:19:48.600977 27298 consensus_queue.cc:260] T 6b990107062e43149cf9e75a7caf1331 P eee2a1bf2d36431099c251ec52579111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df90bb4f0524b3a874e8ac362b8c9f0" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 44473 } } peers { permanent_uuid: "93565a7b2f784d6bb32de5e92acba59b" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 43203 } } peers { permanent_uuid: "eee2a1bf2d36431099c251ec52579111" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39115 } }
I20260926 15:19:48.602010 27278 heartbeater.cc:503] Master 127.6.209.254:35167 was elected leader, sending a full tablet report...
I20260926 15:19:48.602792 27298 ts_tablet_manager.cc:1428] T 6b990107062e43149cf9e75a7caf1331 P eee2a1bf2d36431099c251ec52579111: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 15:19:48.605494 27017 heartbeater.cc:503] Master 127.6.209.254:35167 was elected leader, sending a full tablet report...
I20260926 15:19:48.605926 27296 ts_tablet_manager.cc:1428] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0: Time spent starting tablet: real 0.018s user 0.006s sys 0.012s
W20260926 15:19:48.681416 27018 tablet.cc:2369] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:19:48.743960 27279 tablet.cc:2369] T 6b990107062e43149cf9e75a7caf1331 P eee2a1bf2d36431099c251ec52579111: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:19:48.839407 27149 tablet.cc:2369] T 6b990107062e43149cf9e75a7caf1331 P 93565a7b2f784d6bb32de5e92acba59b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:19:48.911748 27303 raft_consensus.cc:482] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:19:48.912063 27303 raft_consensus.cc:504] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df90bb4f0524b3a874e8ac362b8c9f0" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 44473 } } peers { permanent_uuid: "93565a7b2f784d6bb32de5e92acba59b" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 43203 } } peers { permanent_uuid: "eee2a1bf2d36431099c251ec52579111" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39115 } }
I20260926 15:19:48.913610 27303 leader_election.cc:288] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93565a7b2f784d6bb32de5e92acba59b (127.6.209.194:43203), eee2a1bf2d36431099c251ec52579111 (127.6.209.195:39115)
I20260926 15:19:48.919761 27293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:48.913964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:51793 (local address 127.6.209.194:43203)
0926 15:19:48.914174 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:48.914182 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:48.914656 (+ 474us) server_negotiation.cc:407] Connection header received
0926 15:19:48.914859 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:48.914867 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:48.914984 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:48.915086 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:48.915707 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.916396 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.918081 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.918771 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.918927 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:48.919030 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:48.919365 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 15:19:48.919504 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 15:19:48.919895 27305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:48.913963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.194:43203 (local address 127.6.209.193:51793)
0926 15:19:48.914514 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:48.914528 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:48.914707 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:48.915218 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:48.915223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:48.915259 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:48.915560 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:48.915568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.916536 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.916541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:48.917921 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:48.917932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.918945 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.918950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:48.919141 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:48.919164 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 15:19:48.919335 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 15:19:48.919460 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"spinlock_wait_cycles":15744,"thread_start_us":318,"threads_started":1}
I20260926 15:19:48.920665 27306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:48.914661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.195:39115 (local address 127.6.209.193:47955)
0926 15:19:48.915254 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:48.915267 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:19:48.915419 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:19:48.915722 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:19:48.915726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:19:48.915746 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:19:48.916103 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:48.916110 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.917082 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.917087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:48.918307 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:19:48.918329 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.919975 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.919985 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:19:48.920168 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:48.920189 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 15:19:48.920260 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 15:19:48.920385 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":344,"threads_started":1}
I20260926 15:19:48.920704 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:48.914811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.209.193:47955 (local address 127.6.209.195:39115)
0926 15:19:48.915015 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:48.915023 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:48.915348 (+ 325us) server_negotiation.cc:407] Connection header received
0926 15:19:48.915553 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:48.915558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:48.915610 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:48.915708 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:19:48.916240 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.916961 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.918472 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:48.919551 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:48.919688 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:48.919751 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:48.920358 (+ 607us) server_negotiation.cc:299] Negotiation successful
0926 15:19:48.920513 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 15:19:48.921190 27103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b990107062e43149cf9e75a7caf1331" candidate_uuid: "6df90bb4f0524b3a874e8ac362b8c9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93565a7b2f784d6bb32de5e92acba59b" is_pre_election: true
I20260926 15:19:48.921641 27103 raft_consensus.cc:2454] T 6b990107062e43149cf9e75a7caf1331 P 93565a7b2f784d6bb32de5e92acba59b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6df90bb4f0524b3a874e8ac362b8c9f0 in term 0.
I20260926 15:19:48.921839 27233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b990107062e43149cf9e75a7caf1331" candidate_uuid: "6df90bb4f0524b3a874e8ac362b8c9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee2a1bf2d36431099c251ec52579111" is_pre_election: true
I20260926 15:19:48.922261 26905 leader_election.cc:302] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6df90bb4f0524b3a874e8ac362b8c9f0, 93565a7b2f784d6bb32de5e92acba59b; no voters:
I20260926 15:19:48.922333 27233 raft_consensus.cc:2454] T 6b990107062e43149cf9e75a7caf1331 P eee2a1bf2d36431099c251ec52579111 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6df90bb4f0524b3a874e8ac362b8c9f0 in term 0.
I20260926 15:19:48.922634 27303 raft_consensus.cc:2781] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:19:48.922801 27303 raft_consensus.cc:482] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:19:48.922909 27303 raft_consensus.cc:3037] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:48.925041 27303 raft_consensus.cc:504] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df90bb4f0524b3a874e8ac362b8c9f0" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 44473 } } peers { permanent_uuid: "93565a7b2f784d6bb32de5e92acba59b" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 43203 } } peers { permanent_uuid: "eee2a1bf2d36431099c251ec52579111" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39115 } }
I20260926 15:19:48.925590 27303 leader_election.cc:288] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0 [CANDIDATE]: Term 1 election: Requested vote from peers 93565a7b2f784d6bb32de5e92acba59b (127.6.209.194:43203), eee2a1bf2d36431099c251ec52579111 (127.6.209.195:39115)
I20260926 15:19:48.926033 27103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b990107062e43149cf9e75a7caf1331" candidate_uuid: "6df90bb4f0524b3a874e8ac362b8c9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93565a7b2f784d6bb32de5e92acba59b"
I20260926 15:19:48.926098 27233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b990107062e43149cf9e75a7caf1331" candidate_uuid: "6df90bb4f0524b3a874e8ac362b8c9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee2a1bf2d36431099c251ec52579111"
I20260926 15:19:48.926236 27103 raft_consensus.cc:3037] T 6b990107062e43149cf9e75a7caf1331 P 93565a7b2f784d6bb32de5e92acba59b [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:48.926255 27233 raft_consensus.cc:3037] T 6b990107062e43149cf9e75a7caf1331 P eee2a1bf2d36431099c251ec52579111 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:48.929255 27233 raft_consensus.cc:2454] T 6b990107062e43149cf9e75a7caf1331 P eee2a1bf2d36431099c251ec52579111 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6df90bb4f0524b3a874e8ac362b8c9f0 in term 1.
I20260926 15:19:48.929255 27103 raft_consensus.cc:2454] T 6b990107062e43149cf9e75a7caf1331 P 93565a7b2f784d6bb32de5e92acba59b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6df90bb4f0524b3a874e8ac362b8c9f0 in term 1.
I20260926 15:19:48.929670 26905 leader_election.cc:302] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6df90bb4f0524b3a874e8ac362b8c9f0, 93565a7b2f784d6bb32de5e92acba59b; no voters:
I20260926 15:19:48.929946 27303 raft_consensus.cc:2781] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:19:48.930371 27303 raft_consensus.cc:686] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 6df90bb4f0524b3a874e8ac362b8c9f0, State: Running, Role: LEADER
I20260926 15:19:48.930892 27303 consensus_queue.cc:237] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df90bb4f0524b3a874e8ac362b8c9f0" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 44473 } } peers { permanent_uuid: "93565a7b2f784d6bb32de5e92acba59b" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 43203 } } peers { permanent_uuid: "eee2a1bf2d36431099c251ec52579111" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39115 } }
I20260926 15:19:48.934940 26831 catalog_manager.cc:5495] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6df90bb4f0524b3a874e8ac362b8c9f0 (127.6.209.193). New cstate: current_term: 1 leader_uuid: "6df90bb4f0524b3a874e8ac362b8c9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6df90bb4f0524b3a874e8ac362b8c9f0" member_type: VOTER last_known_addr { host: "127.6.209.193" port: 44473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93565a7b2f784d6bb32de5e92acba59b" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 43203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eee2a1bf2d36431099c251ec52579111" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39115 } health_report { overall_health: UNKNOWN } } }
I20260926 15:19:49.018181 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:49.013493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58118 (local address 127.6.209.193:44473)
0926 15:19:49.013859 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:49.013863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:49.013890 (+ 27us) server_negotiation.cc:407] Connection header received
0926 15:19:49.013964 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:49.013968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:49.014043 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:49.014210 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:19:49.014481 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:49.015085 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:49.016285 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:49.016638 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:49.016718 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:49.017162 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:19:49.017620 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:19:49.017745 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:49.017831 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 15:19:49.017885 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255}
I20260926 15:19:49.026250 27103 raft_consensus.cc:1261] T 6b990107062e43149cf9e75a7caf1331 P 93565a7b2f784d6bb32de5e92acba59b [term 1 FOLLOWER]: Refusing update from remote peer 6df90bb4f0524b3a874e8ac362b8c9f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:19:49.026257 27233 raft_consensus.cc:1261] T 6b990107062e43149cf9e75a7caf1331 P eee2a1bf2d36431099c251ec52579111 [term 1 FOLLOWER]: Refusing update from remote peer 6df90bb4f0524b3a874e8ac362b8c9f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:19:49.027124 27307 consensus_queue.cc:1035] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93565a7b2f784d6bb32de5e92acba59b" member_type: VOTER last_known_addr { host: "127.6.209.194" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:19:49.027478 27303 consensus_queue.cc:1035] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eee2a1bf2d36431099c251ec52579111" member_type: VOTER last_known_addr { host: "127.6.209.195" port: 39115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:19:49.037109 27310 mvcc.cc:205] Tried to move back new op lower bound from 7333625811041607680 to 7333625810664394752. Current Snapshot: MvccSnapshot[applied={T|T < 7333625811041607680}]
I20260926 15:19:49.050619 26830 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:19:49.050834 26830 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:19:49.054139 26830 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b7d0714d0fc4ca0b4cb21b236aa85e2: Sending DeleteTablet for 3 replicas of tablet 6b990107062e43149cf9e75a7caf1331
I20260926 15:19:49.055022 26952 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b990107062e43149cf9e75a7caf1331 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:19:49 UTC) from {username='slave'} at 127.0.0.1:58108
I20260926 15:19:49.055119 27083 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b990107062e43149cf9e75a7caf1331 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:19:49 UTC) from {username='slave'} at 127.0.0.1:32830
I20260926 15:19:49.055881 27213 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b990107062e43149cf9e75a7caf1331 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:19:49 UTC) from {username='slave'} at 127.0.0.1:51388
I20260926 15:19:49.056638 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 26890
I20260926 15:19:49.065028 27322 tablet_replica.cc:330] T 6b990107062e43149cf9e75a7caf1331 P eee2a1bf2d36431099c251ec52579111: stopping tablet replica
I20260926 15:19:49.065470 27322 raft_consensus.cc:2229] T 6b990107062e43149cf9e75a7caf1331 P eee2a1bf2d36431099c251ec52579111 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:19:49.065938 27322 raft_consensus.cc:2258] T 6b990107062e43149cf9e75a7caf1331 P eee2a1bf2d36431099c251ec52579111 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:19:49.069649 27322 ts_tablet_manager.cc:1905] T 6b990107062e43149cf9e75a7caf1331 P eee2a1bf2d36431099c251ec52579111: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:19:49.076355 27320 tablet_replica.cc:330] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0: stopping tablet replica
I20260926 15:19:49.076534 27321 tablet_replica.cc:330] T 6b990107062e43149cf9e75a7caf1331 P 93565a7b2f784d6bb32de5e92acba59b: stopping tablet replica
I20260926 15:19:49.076894 27321 raft_consensus.cc:2229] T 6b990107062e43149cf9e75a7caf1331 P 93565a7b2f784d6bb32de5e92acba59b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:19:49.077316 27321 raft_consensus.cc:2258] T 6b990107062e43149cf9e75a7caf1331 P 93565a7b2f784d6bb32de5e92acba59b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:19:49.078374 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:49.057998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58122 (local address 127.6.209.193:44473)
0926 15:19:49.058131 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:49.058136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:49.070517 (+ 12381us) server_negotiation.cc:407] Connection header received
0926 15:19:49.070885 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:49.070891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:49.070969 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:49.071031 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:49.071617 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:49.072773 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:49.074558 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:49.075158 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:49.075242 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:49.076883 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:49.076893 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:49.076906 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:49.077582 (+ 676us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:49.077641 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:49.077662 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:49.077733 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:49.077954 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 15:19:49.078041 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 15:19:49.078619 27320 raft_consensus.cc:2229] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:19:49.079236 27320 raft_consensus.cc:2258] T 6b990107062e43149cf9e75a7caf1331 P 6df90bb4f0524b3a874e8ac362b8c9f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:19:49.079324 27012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:49.079689 27321 ts_tablet_manager.cc:1905] T 6b990107062e43149cf9e75a7caf1331 P 93565a7b2f784d6bb32de5e92acba59b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:19:49.081997 27322 ts_tablet_manager.cc:1918] T 6b990107062e43149cf9e75a7caf1331 P eee2a1bf2d36431099c251ec52579111: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:19:49.082183 27322 log.cc:1196] T 6b990107062e43149cf9e75a7caf1331 P eee2a1bf2d36431099c251ec52579111: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-2/wal/wals/6b990107062e43149cf9e75a7caf1331
I20260926 15:19:49.082680 27322 ts_tablet_manager.cc:1939] T 6b990107062e43149cf9e75a7caf1331 P eee2a1bf2d36431099c251ec52579111: Deleting consensus metadata
I20260926 15:19:49.084290 26815 catalog_manager.cc:4841] TS eee2a1bf2d36431099c251ec52579111 (127.6.209.195:39115): tablet 6b990107062e43149cf9e75a7caf1331 (table test-table [id=b995ace9fbdb4d66a1d07f19fdff7a7b]) successfully deleted
I20260926 15:19:49.086917 27321 ts_tablet_manager.cc:1918] T 6b990107062e43149cf9e75a7caf1331 P 93565a7b2f784d6bb32de5e92acba59b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:19:49.087031 27321 log.cc:1196] T 6b990107062e43149cf9e75a7caf1331 P 93565a7b2f784d6bb32de5e92acba59b: Deleting WAL directory at /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435694426955-6983-0/minicluster-data/ts-1/wal/wals/6b990107062e43149cf9e75a7caf1331
I20260926 15:19:49.087476 27321 ts_tablet_manager.cc:1939] T 6b990107062e43149cf9e75a7caf1331 P 93565a7b2f784d6bb32de5e92acba59b: Deleting consensus metadata
I20260926 15:19:49.089035 26815 catalog_manager.cc:4841] TS 93565a7b2f784d6bb32de5e92acba59b (127.6.209.194:43203): tablet 6b990107062e43149cf9e75a7caf1331 (table test-table [id=b995ace9fbdb4d66a1d07f19fdff7a7b]) successfully deleted
I20260926 15:19:49.168385 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 26890
W20260926 15:19:49.185025 26818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.209.193:44473 (error 108)
I20260926 15:19:49.185627 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 27021
I20260926 15:19:49.186869 27326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:49.186110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:44473 (local address 127.0.0.1:58132)
0926 15:19:49.186523 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:49.186674 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:44473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":202,"threads_started":1}
W20260926 15:19:49.187136 26818 catalog_manager.cc:4584] TS 6df90bb4f0524b3a874e8ac362b8c9f0 (127.6.209.193:44473): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b990107062e43149cf9e75a7caf1331: Network error: Client connection negotiation failed: client connection to 127.6.209.193:44473: connect: Connection refused (error 111)
I20260926 15:19:49.191900 27293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:49.186295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32834 (local address 127.6.209.194:43203)
0926 15:19:49.186509 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:49.186515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:49.186600 (+ 85us) server_negotiation.cc:407] Connection header received
0926 15:19:49.186769 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:49.186773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:49.186875 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:49.186987 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:49.187439 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:49.188132 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:49.189340 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:49.189703 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:49.189838 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:49.190697 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:49.190705 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:49.190717 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:49.191237 (+ 520us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:49.191280 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:49.191298 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:49.191416 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:49.191531 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 15:19:49.191625 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 15:19:49.192519 27143 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:49.240742 27326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:49.240261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:44473 (local address 127.0.0.1:58148)
0926 15:19:49.240486 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:49.240584 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:44473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 15:19:49.241827 27326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:49.241432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:44473 (local address 127.0.0.1:58162)
0926 15:19:49.241594 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:49.241674 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:44473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 15:19:49.275943 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 27021
I20260926 15:19:49.291510 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 27151
I20260926 15:19:49.297291 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:49.292177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51394 (local address 127.6.209.195:39115)
0926 15:19:49.292337 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:49.292341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:49.292357 (+ 16us) server_negotiation.cc:407] Connection header received
0926 15:19:49.292424 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:49.292427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:49.292525 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:49.292600 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:49.293023 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:49.293728 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:49.294823 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:49.295165 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:49.295244 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:49.296060 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:49.296067 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:49.296081 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:49.296613 (+ 532us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:49.296653 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:49.296670 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:49.296747 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:49.296987 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 15:19:49.297068 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 15:19:49.297896 27273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:49.302122 27326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:49.301607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:44473 (local address 127.0.0.1:58170)
0926 15:19:49.301878 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:49.301964 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:44473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260926 15:19:49.303032 27326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:49.302715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:44473 (local address 127.0.0.1:58172)
0926 15:19:49.302854 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:49.302923 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:44473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 15:19:49.377817 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 27151
I20260926 15:19:49.383550 27326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:49.382981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:44473 (local address 127.0.0.1:58180)
0926 15:19:49.383243 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:49.383353 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:44473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 15:19:49.384594 27326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:49.384151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.193:44473 (local address 127.0.0.1:58182)
0926 15:19:49.384361 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 15:19:49.384441 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.193:44473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 15:19:49.393281 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 26800
I20260926 15:19:49.394196 26861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:49.491101 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 26800
2026-09-26T15:19:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2791 ms)
[----------] 6 tests from EncryptionPolicyTest (19754 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:19:49 dist-test-slave-j6q7 krb5kdc[27332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:19:49 dist-test-slave-j6q7 krb5kdc[27332](info): set up 2 sockets
Sep 26 15:19:49 dist-test-slave-j6q7 krb5kdc[27332](info): commencing operation
krb5kdc: starting...
W20260926 15:19:50.509290 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:19:50 dist-test-slave-j6q7 krb5kdc[27332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:19:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:19:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:19:50.715586 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37601
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:19:51.011653 27345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:51.011904 27345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:51.011991 27345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:51.018133 27345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:19:51.018215 27345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:51.018254 27345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:51.018308 27345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:19:51.018348 27345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:19:51.027001 27345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37601
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:51.028024 27345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:51.029153 27345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:51.039077 27350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:51.039139 27351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:51.039119 27353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:51.040059 27345 server_base.cc:1034] running on GCE node
I20260926 15:19:51.040735 27345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:51.041922 27345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:51.043133 27345 hybrid_clock.cc:648] HybridClock initialized: now 1790435991043094 us; error 27 us; skew 500 ppm
I20260926 15:19:51.043457 27345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:19:51 dist-test-slave-j6q7 krb5kdc[27332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:19:51.046309 27345 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:19:51.047483 27345 webserver.cc:460] Webserver started at http://127.0.0.1:44255/ using document root <none> and password file <none>
I20260926 15:19:51.048004 27345 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:51.048087 27345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:51.048292 27345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:51.050655 27345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "fe1b723e6bdd46c090696be3651ad239"
format_stamp: "Formatted at 2026-09-26 15:19:51 on dist-test-slave-j6q7"
server_key: "881537b6682bd38b6e3f9c77ff8fe95f"
server_key_iv: "a6389af4de5ab2685503264bc8329509"
server_key_version: "encryptionkey@0"
I20260926 15:19:51.051298 27345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "fe1b723e6bdd46c090696be3651ad239"
format_stamp: "Formatted at 2026-09-26 15:19:51 on dist-test-slave-j6q7"
server_key: "881537b6682bd38b6e3f9c77ff8fe95f"
server_key_iv: "a6389af4de5ab2685503264bc8329509"
server_key_version: "encryptionkey@0"
I20260926 15:19:51.055667 27345 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 15:19:51.058367 27360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:51.059394 27345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 15:19:51.059510 27345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "fe1b723e6bdd46c090696be3651ad239"
format_stamp: "Formatted at 2026-09-26 15:19:51 on dist-test-slave-j6q7"
server_key: "881537b6682bd38b6e3f9c77ff8fe95f"
server_key_iv: "a6389af4de5ab2685503264bc8329509"
server_key_version: "encryptionkey@0"
I20260926 15:19:51.059635 27345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:51.070643 27345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:51.073932 27345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:51.074147 27345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:51.090113 27345 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37601
I20260926 15:19:51.090143 27411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37601 every 8 connection(s)
I20260926 15:19:51.091276 27345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:19:51.093713 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 27345
I20260926 15:19:51.093933 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:19:51.094692 6983 external_mini_cluster.cc:1428] Setting key a23f1d9c4201f9a14415b65dd5a5c375
I20260926 15:19:51.094998 27412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:51.107890 27412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe1b723e6bdd46c090696be3651ad239: Bootstrap starting.
Sep 26 15:19:51 dist-test-slave-j6q7 krb5kdc[27332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 15:19:51.110827 27412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe1b723e6bdd46c090696be3651ad239: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:51.111820 27412 log.cc:824] T 00000000000000000000000000000000 P fe1b723e6bdd46c090696be3651ad239: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:51.114339 27412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe1b723e6bdd46c090696be3651ad239: No bootstrap required, opened a new log
I20260926 15:19:51.115710 27415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:51.096266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60783 (local address 127.0.0.1:37601)
0926 15:19:51.096970 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:51.096979 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:51.097014 (+ 35us) server_negotiation.cc:407] Connection header received
0926 15:19:51.098978 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:51.098997 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:51.099318 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:51.099582 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:51.101230 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:51.102376 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:51.104139 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:51.104881 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:51.105011 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:51.109123 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:51.109129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:51.109139 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:51.109170 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:51.111881 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:51.112663 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:51.112669 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:51.112674 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:51.112911 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:51.113376 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:51.113381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:51.113385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:51.113860 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:51.114154 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:51.114637 (+ 483us) server_negotiation.cc:299] Negotiation successful
0926 15:19:51.114936 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":338,"threads_started":1}
I20260926 15:19:51.119230 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:19:51.122018 27412 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe1b723e6bdd46c090696be3651ad239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1b723e6bdd46c090696be3651ad239" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37601 } }
I20260926 15:19:51.122355 27412 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe1b723e6bdd46c090696be3651ad239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:51.122469 27412 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe1b723e6bdd46c090696be3651ad239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe1b723e6bdd46c090696be3651ad239, State: Initialized, Role: FOLLOWER
I20260926 15:19:51.123015 27412 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe1b723e6bdd46c090696be3651ad239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1b723e6bdd46c090696be3651ad239" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37601 } }
I20260926 15:19:51.123235 27412 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe1b723e6bdd46c090696be3651ad239 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:19:51.123381 27412 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe1b723e6bdd46c090696be3651ad239 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:19:51.123579 27412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe1b723e6bdd46c090696be3651ad239 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:51.126696 27412 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe1b723e6bdd46c090696be3651ad239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1b723e6bdd46c090696be3651ad239" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37601 } }
I20260926 15:19:51.127130 27412 leader_election.cc:302] T 00000000000000000000000000000000 P fe1b723e6bdd46c090696be3651ad239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe1b723e6bdd46c090696be3651ad239; no voters:
I20260926 15:19:51.127728 27412 leader_election.cc:288] T 00000000000000000000000000000000 P fe1b723e6bdd46c090696be3651ad239 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:19:51.127808 27422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe1b723e6bdd46c090696be3651ad239 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:19:51.128872 27422 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe1b723e6bdd46c090696be3651ad239 [term 1 LEADER]: Becoming Leader. State: Replica: fe1b723e6bdd46c090696be3651ad239, State: Running, Role: LEADER
I20260926 15:19:51.129464 27422 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe1b723e6bdd46c090696be3651ad239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1b723e6bdd46c090696be3651ad239" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37601 } }
I20260926 15:19:51.129812 27412 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe1b723e6bdd46c090696be3651ad239 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:19:51.134507 27424 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe1b723e6bdd46c090696be3651ad239 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe1b723e6bdd46c090696be3651ad239. Latest consensus state: current_term: 1 leader_uuid: "fe1b723e6bdd46c090696be3651ad239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1b723e6bdd46c090696be3651ad239" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37601 } } }
I20260926 15:19:51.134924 27424 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe1b723e6bdd46c090696be3651ad239 [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:51.135610 27423 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe1b723e6bdd46c090696be3651ad239 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe1b723e6bdd46c090696be3651ad239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1b723e6bdd46c090696be3651ad239" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37601 } } }
I20260926 15:19:51.135921 27423 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe1b723e6bdd46c090696be3651ad239 [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:51.140241 27415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:51.123801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60731 (local address 127.0.0.1:37601)
0926 15:19:51.123957 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:51.123962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:51.124410 (+ 448us) server_negotiation.cc:407] Connection header received
0926 15:19:51.124546 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:51.124549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:51.124594 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:51.124661 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:51.126470 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:51.127202 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:51.128663 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:51.129269 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:51.129408 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:51.131820 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:51.131831 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:51.131835 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:51.131858 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:51.135458 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:51.136539 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:51.136547 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:51.136552 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:51.136694 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:51.137337 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:51.137345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:51.137350 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:51.137853 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:51.139314 (+ 1461us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:51.139500 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 15:19:51.139697 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 15:19:51.144800 27431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:19:51.149150 27431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:19:51.157276 27431 catalog_manager.cc:1347] Generated new cluster ID: 59a54722d9af407ab7459303730c03f2
I20260926 15:19:51.157361 27431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:19:51.164597 27431 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:19:51.165584 27431 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:19:51.169734 27431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe1b723e6bdd46c090696be3651ad239: Generated new TSK 0
I20260926 15:19:51.170289 27431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:19:51.177098 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 27345
I20260926 15:19:51.178092 27407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:51.255326 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 27345
W20260926 15:19:51.271214 27330 connection.cc:535] client connection to 127.0.0.1:37601 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37601): Connection reset by peer (error 104)
2026-09-26T15:19:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1755 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:19:51 dist-test-slave-j6q7 krb5kdc[27445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:19:51 dist-test-slave-j6q7 krb5kdc[27445](info): set up 2 sockets
Sep 26 15:19:51 dist-test-slave-j6q7 krb5kdc[27445](info): commencing operation
krb5kdc: starting...
W20260926 15:19:52.252887 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:19:52 dist-test-slave-j6q7 krb5kdc[27445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:19:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:19:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:19:52.443624 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:19:52.735190 27458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:52.735555 27458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:52.735724 27458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:52.742303 27458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:19:52.742383 27458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:52.742424 27458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:52.742502 27458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:19:52.742560 27458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:19:52.751181 27458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46705
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:52.752205 27458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:52.753440 27458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:52.763844 27463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:52.763878 27464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:52.763780 27466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:52.764988 27458 server_base.cc:1034] running on GCE node
I20260926 15:19:52.765648 27458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:52.766772 27458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:52.767982 27458 hybrid_clock.cc:648] HybridClock initialized: now 1790435992767945 us; error 45 us; skew 500 ppm
I20260926 15:19:52.768262 27458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:19:52 dist-test-slave-j6q7 krb5kdc[27445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:19:52.770987 27458 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:19:52.772085 27458 webserver.cc:460] Webserver started at http://127.0.0.1:46231/ using document root <none> and password file <none>
I20260926 15:19:52.772593 27458 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:52.772677 27458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:52.773010 27458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:52.775499 27458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "7673619ed621465f84ef7fccce29677b"
format_stamp: "Formatted at 2026-09-26 15:19:52 on dist-test-slave-j6q7"
server_key: "9219a47127ba8aa86c82ec59d6f8dc89"
server_key_iv: "ff895a62a74cc2bb98c187c3290e20c4"
server_key_version: "encryptionkey@0"
I20260926 15:19:52.776079 27458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "7673619ed621465f84ef7fccce29677b"
format_stamp: "Formatted at 2026-09-26 15:19:52 on dist-test-slave-j6q7"
server_key: "9219a47127ba8aa86c82ec59d6f8dc89"
server_key_iv: "ff895a62a74cc2bb98c187c3290e20c4"
server_key_version: "encryptionkey@0"
I20260926 15:19:52.780385 27458 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260926 15:19:52.783457 27473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:52.784611 27458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:19:52.784756 27458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "7673619ed621465f84ef7fccce29677b"
format_stamp: "Formatted at 2026-09-26 15:19:52 on dist-test-slave-j6q7"
server_key: "9219a47127ba8aa86c82ec59d6f8dc89"
server_key_iv: "ff895a62a74cc2bb98c187c3290e20c4"
server_key_version: "encryptionkey@0"
I20260926 15:19:52.784968 27458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:52.798027 27458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:52.801863 27458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:52.802098 27458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:52.818307 27458 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46705
I20260926 15:19:52.818374 27524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46705 every 8 connection(s)
I20260926 15:19:52.819470 27458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:19:52.822836 27525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:52.829691 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 27458
I20260926 15:19:52.829864 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:19:52.830341 6983 external_mini_cluster.cc:1428] Setting key b8338e5b0d90a08246a8c673fcd2f6a3
I20260926 15:19:52.832288 27525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7673619ed621465f84ef7fccce29677b: Bootstrap starting.
I20260926 15:19:52.835183 27525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7673619ed621465f84ef7fccce29677b: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:52.836351 27525 log.cc:824] T 00000000000000000000000000000000 P 7673619ed621465f84ef7fccce29677b: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:52.838580 27525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7673619ed621465f84ef7fccce29677b: No bootstrap required, opened a new log
I20260926 15:19:52.842413 27528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:52.831882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55189 (local address 127.0.0.1:46705)
0926 15:19:52.832593 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:52.832603 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:52.832639 (+ 36us) server_negotiation.cc:407] Connection header received
0926 15:19:52.834326 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:52.834344 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:52.834636 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:52.834931 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:52.835593 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:52.837235 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:52.838416 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:52.839028 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:52.839208 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:52.840357 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:52.840365 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:52.840376 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:52.840936 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:52.841011 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:52.841049 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:52.841218 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:52.841478 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 15:19:52.841613 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":332,"threads_started":1}
I20260926 15:19:52.845801 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:19:52.846503 27525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7673619ed621465f84ef7fccce29677b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7673619ed621465f84ef7fccce29677b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46705 } }
I20260926 15:19:52.846805 27525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7673619ed621465f84ef7fccce29677b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:52.846894 27525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7673619ed621465f84ef7fccce29677b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7673619ed621465f84ef7fccce29677b, State: Initialized, Role: FOLLOWER
I20260926 15:19:52.847357 27525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7673619ed621465f84ef7fccce29677b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7673619ed621465f84ef7fccce29677b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46705 } }
I20260926 15:19:52.847558 27525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7673619ed621465f84ef7fccce29677b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:19:52.847702 27525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7673619ed621465f84ef7fccce29677b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:19:52.847867 27525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7673619ed621465f84ef7fccce29677b [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:52.850086 27525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7673619ed621465f84ef7fccce29677b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7673619ed621465f84ef7fccce29677b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46705 } }
I20260926 15:19:52.850605 27525 leader_election.cc:302] T 00000000000000000000000000000000 P 7673619ed621465f84ef7fccce29677b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7673619ed621465f84ef7fccce29677b; no voters:
I20260926 15:19:52.851368 27525 leader_election.cc:288] T 00000000000000000000000000000000 P 7673619ed621465f84ef7fccce29677b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:19:52.851468 27535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7673619ed621465f84ef7fccce29677b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:19:52.853117 27535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7673619ed621465f84ef7fccce29677b [term 1 LEADER]: Becoming Leader. State: Replica: 7673619ed621465f84ef7fccce29677b, State: Running, Role: LEADER
I20260926 15:19:52.853408 27525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7673619ed621465f84ef7fccce29677b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:19:52.853686 27535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7673619ed621465f84ef7fccce29677b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7673619ed621465f84ef7fccce29677b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46705 } }
I20260926 15:19:52.857373 27528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:52.849801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48175 (local address 127.0.0.1:46705)
0926 15:19:52.849965 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:52.849971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:52.850268 (+ 297us) server_negotiation.cc:407] Connection header received
0926 15:19:52.850420 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:52.850423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:52.850465 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:52.850529 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:52.851128 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:52.852359 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:52.854295 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:52.854916 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:52.855060 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:52.856450 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:52.856458 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:52.856464 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:52.856486 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:52.856557 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:52.856580 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:52.856704 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:52.856926 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 15:19:52.856982 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 15:19:52.861992 27537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7673619ed621465f84ef7fccce29677b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7673619ed621465f84ef7fccce29677b. Latest consensus state: current_term: 1 leader_uuid: "7673619ed621465f84ef7fccce29677b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7673619ed621465f84ef7fccce29677b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46705 } } }
I20260926 15:19:52.862287 27537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7673619ed621465f84ef7fccce29677b [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:52.862110 27536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7673619ed621465f84ef7fccce29677b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7673619ed621465f84ef7fccce29677b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7673619ed621465f84ef7fccce29677b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46705 } } }
I20260926 15:19:52.862476 27536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7673619ed621465f84ef7fccce29677b [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:52.863096 27545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:19:52.871791 27545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:19:52.882397 27545 catalog_manager.cc:1347] Generated new cluster ID: eb50161451714bb2b25f0e79099ce8df
I20260926 15:19:52.882512 27545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:19:52.907946 27545 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:19:52.908958 27545 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:19:52.916499 27545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7673619ed621465f84ef7fccce29677b: Generated new TSK 0
I20260926 15:19:52.917204 27545 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:19:52.924413 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 27458
W20260926 15:19:52.925360 27520 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55189
W20260926 15:19:52.926050 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 27458: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:19:52.926229 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 27458
W20260926 15:19:52.941646 27443 connection.cc:535] client connection to 127.0.0.1:46705 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46705): Connection reset by peer (error 104)
2026-09-26T15:19:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1669 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:19:53 dist-test-slave-j6q7 krb5kdc[27556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:19:53 dist-test-slave-j6q7 krb5kdc[27556](info): set up 2 sockets
Sep 26 15:19:53 dist-test-slave-j6q7 krb5kdc[27556](info): commencing operation
krb5kdc: starting...
W20260926 15:19:53.937022 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:19:54 dist-test-slave-j6q7 krb5kdc[27556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:19:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:19:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:19:54.137751 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41359
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:19:54.429531 27569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:54.429766 27569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:54.429910 27569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:54.436261 27569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:19:54.436358 27569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:54.436399 27569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:54.436468 27569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:19:54.436532 27569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:19:54.445271 27569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41359
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:54.446291 27569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:54.447350 27569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:54.458051 27574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:54.458118 27577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:54.458041 27575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:54.458527 27569 server_base.cc:1034] running on GCE node
I20260926 15:19:54.459333 27569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:54.460522 27569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:54.461731 27569 hybrid_clock.cc:648] HybridClock initialized: now 1790435994461698 us; error 46 us; skew 500 ppm
I20260926 15:19:54.462024 27569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:19:54 dist-test-slave-j6q7 krb5kdc[27556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:19:54.464923 27569 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:19:54.466144 27569 webserver.cc:460] Webserver started at http://127.0.0.1:34223/ using document root <none> and password file <none>
I20260926 15:19:54.466749 27569 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:54.466898 27569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:54.467161 27569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:54.469702 27569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "6f957cb2db524a2683a42c8024d8197e"
format_stamp: "Formatted at 2026-09-26 15:19:54 on dist-test-slave-j6q7"
server_key: "9cb148d426672fc2a4f8586c280c704c"
server_key_iv: "de604105ac9b1a9779d311085add1944"
server_key_version: "encryptionkey@0"
I20260926 15:19:54.470332 27569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "6f957cb2db524a2683a42c8024d8197e"
format_stamp: "Formatted at 2026-09-26 15:19:54 on dist-test-slave-j6q7"
server_key: "9cb148d426672fc2a4f8586c280c704c"
server_key_iv: "de604105ac9b1a9779d311085add1944"
server_key_version: "encryptionkey@0"
I20260926 15:19:54.474650 27569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 15:19:54.477607 27584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:54.478631 27569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:19:54.478782 27569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "6f957cb2db524a2683a42c8024d8197e"
format_stamp: "Formatted at 2026-09-26 15:19:54 on dist-test-slave-j6q7"
server_key: "9cb148d426672fc2a4f8586c280c704c"
server_key_iv: "de604105ac9b1a9779d311085add1944"
server_key_version: "encryptionkey@0"
I20260926 15:19:54.478948 27569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:54.483741 27569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:54.487576 27569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:54.487830 27569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:54.504547 27569 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41359
I20260926 15:19:54.504619 27635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41359 every 8 connection(s)
I20260926 15:19:54.505817 27569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:19:54.509081 27636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:54.514273 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 27569
I20260926 15:19:54.514449 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:19:54.514837 6983 external_mini_cluster.cc:1428] Setting key b69b62fe0c4d05e88ed2724602265a66
I20260926 15:19:54.519724 27636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f957cb2db524a2683a42c8024d8197e: Bootstrap starting.
I20260926 15:19:54.523204 27639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:54.516618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52575 (local address 127.0.0.1:41359)
0926 15:19:54.517468 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:54.517481 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:54.517526 (+ 45us) server_negotiation.cc:407] Connection header received
0926 15:19:54.519572 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:54.519596 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:54.520047 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:54.520454 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:54.521408 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:54.521419 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:54.521442 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:54.521461 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:54.521537 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:54.521564 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:54.521771 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:54.521898 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 15:19:54.522081 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":378,"threads_started":1}
I20260926 15:19:54.523597 27636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f957cb2db524a2683a42c8024d8197e: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:54.524755 27636 log.cc:824] T 00000000000000000000000000000000 P 6f957cb2db524a2683a42c8024d8197e: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:54.526883 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:19:54.527680 27636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f957cb2db524a2683a42c8024d8197e: No bootstrap required, opened a new log
I20260926 15:19:54.532853 27639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:54.530752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38497 (local address 127.0.0.1:41359)
0926 15:19:54.530906 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:54.530912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:54.531203 (+ 291us) server_negotiation.cc:407] Connection header received
0926 15:19:54.531314 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:54.531320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:54.531371 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:54.531464 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:54.532258 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:54.532263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:54.532268 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:54.532284 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:54.532325 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:54.532346 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:54.532480 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:54.532540 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 15:19:54.532589 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 15:19:54.535465 27636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f957cb2db524a2683a42c8024d8197e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f957cb2db524a2683a42c8024d8197e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41359 } }
I20260926 15:19:54.535768 27636 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f957cb2db524a2683a42c8024d8197e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:54.535862 27636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f957cb2db524a2683a42c8024d8197e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f957cb2db524a2683a42c8024d8197e, State: Initialized, Role: FOLLOWER
I20260926 15:19:54.536388 27636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f957cb2db524a2683a42c8024d8197e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f957cb2db524a2683a42c8024d8197e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41359 } }
I20260926 15:19:54.536594 27636 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f957cb2db524a2683a42c8024d8197e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:19:54.536720 27636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f957cb2db524a2683a42c8024d8197e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:19:54.536890 27636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f957cb2db524a2683a42c8024d8197e [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:54.538995 27636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f957cb2db524a2683a42c8024d8197e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f957cb2db524a2683a42c8024d8197e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41359 } }
I20260926 15:19:54.539469 27636 leader_election.cc:302] T 00000000000000000000000000000000 P 6f957cb2db524a2683a42c8024d8197e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f957cb2db524a2683a42c8024d8197e; no voters:
I20260926 15:19:54.539976 27636 leader_election.cc:288] T 00000000000000000000000000000000 P 6f957cb2db524a2683a42c8024d8197e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:19:54.540076 27646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f957cb2db524a2683a42c8024d8197e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:19:54.540737 27646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f957cb2db524a2683a42c8024d8197e [term 1 LEADER]: Becoming Leader. State: Replica: 6f957cb2db524a2683a42c8024d8197e, State: Running, Role: LEADER
I20260926 15:19:54.541268 27646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f957cb2db524a2683a42c8024d8197e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f957cb2db524a2683a42c8024d8197e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41359 } }
I20260926 15:19:54.541857 27636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f957cb2db524a2683a42c8024d8197e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:19:54.543802 27648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f957cb2db524a2683a42c8024d8197e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f957cb2db524a2683a42c8024d8197e. Latest consensus state: current_term: 1 leader_uuid: "6f957cb2db524a2683a42c8024d8197e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f957cb2db524a2683a42c8024d8197e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41359 } } }
I20260926 15:19:54.544281 27648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f957cb2db524a2683a42c8024d8197e [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:54.545917 27647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f957cb2db524a2683a42c8024d8197e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f957cb2db524a2683a42c8024d8197e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f957cb2db524a2683a42c8024d8197e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41359 } } }
I20260926 15:19:54.546139 27654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:19:54.546231 27647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f957cb2db524a2683a42c8024d8197e [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:54.550534 27654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:19:54.558226 27654 catalog_manager.cc:1347] Generated new cluster ID: 0cc16e16ce3349b1a59665f2fa59e23c
I20260926 15:19:54.558355 27654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:19:54.587885 27654 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:19:54.588229 27654 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:19:54.599814 27654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f957cb2db524a2683a42c8024d8197e: Generated new TSK 0
I20260926 15:19:54.600656 27654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:19:54.618875 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 27569
W20260926 15:19:54.619729 27631 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52575
W20260926 15:19:54.620381 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 27569: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:19:54.620527 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 27569
2026-09-26T15:19:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1694 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:19:54 dist-test-slave-j6q7 krb5kdc[27667](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:19:54 dist-test-slave-j6q7 krb5kdc[27667](info): set up 2 sockets
Sep 26 15:19:54 dist-test-slave-j6q7 krb5kdc[27667](info): commencing operation
krb5kdc: starting...
W20260926 15:19:55.616730 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:19:55 dist-test-slave-j6q7 krb5kdc[27667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:19:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:19:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:19:55.812487 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39437
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:19:56.098739 27680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:56.098959 27680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:56.099056 27680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:56.105564 27680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:19:56.105655 27680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:56.105700 27680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:56.105758 27680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:19:56.105801 27680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:19:56.114636 27680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:56.115618 27680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:56.116662 27680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:56.127590 27685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:56.127554 27686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:56.127615 27688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:56.128535 27680 server_base.cc:1034] running on GCE node
I20260926 15:19:56.129257 27680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:56.130412 27680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:56.131620 27680 hybrid_clock.cc:648] HybridClock initialized: now 1790435996131595 us; error 51 us; skew 500 ppm
I20260926 15:19:56.131975 27680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:19:56 dist-test-slave-j6q7 krb5kdc[27667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:19:56.134887 27680 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:19:56.136116 27680 webserver.cc:460] Webserver started at http://127.0.0.1:38133/ using document root <none> and password file <none>
I20260926 15:19:56.136736 27680 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:56.136886 27680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:56.137145 27680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:56.139721 27680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "a7f57b9e158c4268973fb40c03601876"
format_stamp: "Formatted at 2026-09-26 15:19:56 on dist-test-slave-j6q7"
server_key: "bdfd3881bc7f95d7be75b674545b7fb7"
server_key_iv: "4ed3414c399dcd92e79b1ec80cc76cab"
server_key_version: "encryptionkey@0"
I20260926 15:19:56.140451 27680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "a7f57b9e158c4268973fb40c03601876"
format_stamp: "Formatted at 2026-09-26 15:19:56 on dist-test-slave-j6q7"
server_key: "bdfd3881bc7f95d7be75b674545b7fb7"
server_key_iv: "4ed3414c399dcd92e79b1ec80cc76cab"
server_key_version: "encryptionkey@0"
I20260926 15:19:56.145004 27680 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 15:19:56.147730 27695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:56.148746 27680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:19:56.148996 27680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "a7f57b9e158c4268973fb40c03601876"
format_stamp: "Formatted at 2026-09-26 15:19:56 on dist-test-slave-j6q7"
server_key: "bdfd3881bc7f95d7be75b674545b7fb7"
server_key_iv: "4ed3414c399dcd92e79b1ec80cc76cab"
server_key_version: "encryptionkey@0"
I20260926 15:19:56.149233 27680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:56.169926 27680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:56.173106 27680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:56.173398 27680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:56.190037 27680 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39437
I20260926 15:19:56.190088 27746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39437 every 8 connection(s)
I20260926 15:19:56.191329 27680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:19:56.194693 27747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:56.200079 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 27680
I20260926 15:19:56.200256 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:19:56.200826 6983 external_mini_cluster.cc:1428] Setting key 97d712ab9655bffd945f9c5e7e71559d
I20260926 15:19:56.204246 27747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f57b9e158c4268973fb40c03601876: Bootstrap starting.
I20260926 15:19:56.207779 27747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7f57b9e158c4268973fb40c03601876: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:56.208920 27747 log.cc:824] T 00000000000000000000000000000000 P a7f57b9e158c4268973fb40c03601876: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:56.211313 27747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f57b9e158c4268973fb40c03601876: No bootstrap required, opened a new log
Sep 26 15:19:56 dist-test-slave-j6q7 krb5kdc[27667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 15:19:56.220588 27747 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7f57b9e158c4268973fb40c03601876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f57b9e158c4268973fb40c03601876" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39437 } }
I20260926 15:19:56.221045 27747 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7f57b9e158c4268973fb40c03601876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:56.221187 27747 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7f57b9e158c4268973fb40c03601876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7f57b9e158c4268973fb40c03601876, State: Initialized, Role: FOLLOWER
I20260926 15:19:56.221747 27747 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7f57b9e158c4268973fb40c03601876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f57b9e158c4268973fb40c03601876" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39437 } }
I20260926 15:19:56.221984 27747 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7f57b9e158c4268973fb40c03601876 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:19:56.222121 27747 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7f57b9e158c4268973fb40c03601876 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:19:56.222287 27747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7f57b9e158c4268973fb40c03601876 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:56.223930 27750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:56.202646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60927 (local address 127.0.0.1:39437)
0926 15:19:56.203463 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:56.203478 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:56.203862 (+ 384us) server_negotiation.cc:407] Connection header received
0926 15:19:56.206215 (+ 2353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:56.206239 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:56.206678 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:56.207052 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:56.208213 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:56.209800 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:56.210915 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:56.211845 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:56.212033 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:56.215888 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:56.215896 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:56.215913 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:56.215954 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:56.219770 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:56.220393 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:56.220402 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:56.220412 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:56.220604 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:56.221042 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:56.221047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:56.221052 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:56.221584 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:56.221870 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:56.222268 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 15:19:56.222641 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":329,"threads_started":1}
I20260926 15:19:56.225046 27747 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7f57b9e158c4268973fb40c03601876 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f57b9e158c4268973fb40c03601876" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39437 } }
I20260926 15:19:56.225481 27747 leader_election.cc:302] T 00000000000000000000000000000000 P a7f57b9e158c4268973fb40c03601876 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7f57b9e158c4268973fb40c03601876; no voters:
I20260926 15:19:56.226064 27747 leader_election.cc:288] T 00000000000000000000000000000000 P a7f57b9e158c4268973fb40c03601876 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:19:56.226164 27752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7f57b9e158c4268973fb40c03601876 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:19:56.227587 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:19:56.227764 27752 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7f57b9e158c4268973fb40c03601876 [term 1 LEADER]: Becoming Leader. State: Replica: a7f57b9e158c4268973fb40c03601876, State: Running, Role: LEADER
I20260926 15:19:56.228055 27747 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7f57b9e158c4268973fb40c03601876 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:19:56.228281 27752 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7f57b9e158c4268973fb40c03601876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f57b9e158c4268973fb40c03601876" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39437 } }
I20260926 15:19:56.230870 27753 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f57b9e158c4268973fb40c03601876 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7f57b9e158c4268973fb40c03601876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f57b9e158c4268973fb40c03601876" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39437 } } }
I20260926 15:19:56.231215 27753 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f57b9e158c4268973fb40c03601876 [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:56.230849 27754 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f57b9e158c4268973fb40c03601876 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7f57b9e158c4268973fb40c03601876. Latest consensus state: current_term: 1 leader_uuid: "a7f57b9e158c4268973fb40c03601876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f57b9e158c4268973fb40c03601876" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39437 } } }
I20260926 15:19:56.231824 27754 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f57b9e158c4268973fb40c03601876 [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:56.237291 27759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:19:56.241822 27759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:19:56.257766 27759 catalog_manager.cc:1347] Generated new cluster ID: 6aae601b5dcb499983afcfcbb41088b0
I20260926 15:19:56.257858 27759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:19:56.273438 27759 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:19:56.275015 27750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:56.255304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52671 (local address 127.0.0.1:39437)
0926 15:19:56.255483 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:56.255488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:56.256019 (+ 531us) server_negotiation.cc:407] Connection header received
0926 15:19:56.256168 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:56.256174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:56.256249 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:56.256392 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:56.258664 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:56.259991 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:56.262113 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:56.262964 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:56.263071 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:56.266706 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:56.266716 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:56.266723 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:19:56.266768 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:56.271067 (+ 4299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:56.271969 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:56.271978 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:56.271984 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:56.272177 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:19:56.272699 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:19:56.272706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:19:56.272715 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:19:56.273221 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:56.273525 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:56.274232 (+ 707us) server_negotiation.cc:299] Negotiation successful
0926 15:19:56.274526 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 15:19:56.275156 27759 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:19:56.285627 27759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7f57b9e158c4268973fb40c03601876: Generated new TSK 0
I20260926 15:19:56.286532 27759 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:19:56.295830 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 27680
I20260926 15:19:56.297235 27742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:19:56.376420 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 27680
2026-09-26T15:19:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1756 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:19:56 dist-test-slave-j6q7 krb5kdc[27779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:19:56 dist-test-slave-j6q7 krb5kdc[27779](info): set up 2 sockets
Sep 26 15:19:56 dist-test-slave-j6q7 krb5kdc[27779](info): commencing operation
krb5kdc: starting...
W20260926 15:19:57.425071 6983 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:19:57 dist-test-slave-j6q7 krb5kdc[27779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:19:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:19:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:19:57.626106 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32981
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:19:57.930750 27792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:57.930990 27792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:57.931067 27792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:57.937700 27792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:19:57.937786 27792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:57.937840 27792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:57.937891 27792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:19:57.937934 27792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:19:57.947399 27792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:57.948369 27792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:57.949501 27792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:57.960711 27800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:57.960887 27798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:57.961393 27792 server_base.cc:1034] running on GCE node
W20260926 15:19:57.961505 27797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:57.962188 27792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:57.963434 27792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:57.964651 27792 hybrid_clock.cc:648] HybridClock initialized: now 1790435997964614 us; error 54 us; skew 500 ppm
I20260926 15:19:57.965009 27792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:19:57 dist-test-slave-j6q7 krb5kdc[27779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:19:57.967890 27792 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:19:57.969120 27792 webserver.cc:460] Webserver started at http://127.0.0.1:41541/ using document root <none> and password file <none>
I20260926 15:19:57.969686 27792 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:57.969775 27792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:57.969996 27792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:57.972534 27792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "90dcc6df9e944ee4ae2de3132fde0450"
format_stamp: "Formatted at 2026-09-26 15:19:57 on dist-test-slave-j6q7"
server_key: "e8987f4732a8897935cffbab53e30198"
server_key_iv: "75126f145c932e7278bbeb78b37c1e90"
server_key_version: "encryptionkey@0"
I20260926 15:19:57.973202 27792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "90dcc6df9e944ee4ae2de3132fde0450"
format_stamp: "Formatted at 2026-09-26 15:19:57 on dist-test-slave-j6q7"
server_key: "e8987f4732a8897935cffbab53e30198"
server_key_iv: "75126f145c932e7278bbeb78b37c1e90"
server_key_version: "encryptionkey@0"
I20260926 15:19:57.977801 27792 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:19:57.980720 27807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:57.981915 27792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:19:57.982045 27792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "90dcc6df9e944ee4ae2de3132fde0450"
format_stamp: "Formatted at 2026-09-26 15:19:57 on dist-test-slave-j6q7"
server_key: "e8987f4732a8897935cffbab53e30198"
server_key_iv: "75126f145c932e7278bbeb78b37c1e90"
server_key_version: "encryptionkey@0"
I20260926 15:19:57.982175 27792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:57.992533 27792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:57.996376 27792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:57.996593 27792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:58.013727 27792 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32981
I20260926 15:19:58.013762 27858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32981 every 8 connection(s)
I20260926 15:19:58.014971 27792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:19:58.018075 27859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:58.022013 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 27792
I20260926 15:19:58.022183 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:19:58.022615 6983 external_mini_cluster.cc:1428] Setting key c2b2556d1882a3531fe5d18179c92bb2
I20260926 15:19:58.028040 27859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90dcc6df9e944ee4ae2de3132fde0450: Bootstrap starting.
I20260926 15:19:58.031322 27859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90dcc6df9e944ee4ae2de3132fde0450: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:58.032521 27859 log.cc:824] T 00000000000000000000000000000000 P 90dcc6df9e944ee4ae2de3132fde0450: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:58.034984 27862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:58.024174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38171 (local address 127.0.0.1:32981)
0926 15:19:58.024937 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:58.024950 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:58.024987 (+ 37us) server_negotiation.cc:407] Connection header received
0926 15:19:58.026665 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:58.026683 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:58.027023 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:58.027338 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:58.027962 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:58.029313 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:58.031000 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:58.031632 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:58.031742 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:58.032958 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:58.032965 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:58.032976 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:58.033492 (+ 516us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:58.033567 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:58.033591 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:58.033718 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:58.033934 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 15:19:58.034161 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":387,"threads_started":1}
I20260926 15:19:58.035269 27859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90dcc6df9e944ee4ae2de3132fde0450: No bootstrap required, opened a new log
I20260926 15:19:58.038190 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:19:58.042663 27859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90dcc6df9e944ee4ae2de3132fde0450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90dcc6df9e944ee4ae2de3132fde0450" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32981 } }
I20260926 15:19:58.043099 27859 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90dcc6df9e944ee4ae2de3132fde0450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:58.043217 27859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90dcc6df9e944ee4ae2de3132fde0450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90dcc6df9e944ee4ae2de3132fde0450, State: Initialized, Role: FOLLOWER
I20260926 15:19:58.043911 27859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90dcc6df9e944ee4ae2de3132fde0450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90dcc6df9e944ee4ae2de3132fde0450" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32981 } }
I20260926 15:19:58.044180 27859 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90dcc6df9e944ee4ae2de3132fde0450 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:19:58.044315 27859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90dcc6df9e944ee4ae2de3132fde0450 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:19:58.044493 27859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90dcc6df9e944ee4ae2de3132fde0450 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:58.046876 27859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90dcc6df9e944ee4ae2de3132fde0450 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90dcc6df9e944ee4ae2de3132fde0450" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32981 } }
I20260926 15:19:58.047390 27859 leader_election.cc:302] T 00000000000000000000000000000000 P 90dcc6df9e944ee4ae2de3132fde0450 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90dcc6df9e944ee4ae2de3132fde0450; no voters:
I20260926 15:19:58.048177 27862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:58.041970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44665 (local address 127.0.0.1:32981)
0926 15:19:58.042107 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:58.042111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:58.042595 (+ 484us) server_negotiation.cc:407] Connection header received
0926 15:19:58.042772 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:58.042776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:58.042845 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:58.042939 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:58.043489 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:58.044592 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:58.045755 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:19:58.046378 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:19:58.046494 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:19:58.047364 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:58.047372 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:58.047381 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:58.047402 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:58.047443 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:58.047461 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:58.047547 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:58.047684 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 15:19:58.047768 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 15:19:58.048601 27859 leader_election.cc:288] T 00000000000000000000000000000000 P 90dcc6df9e944ee4ae2de3132fde0450 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:19:58.048702 27869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90dcc6df9e944ee4ae2de3132fde0450 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:19:58.050907 27859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90dcc6df9e944ee4ae2de3132fde0450 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:19:58.053148 27869 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90dcc6df9e944ee4ae2de3132fde0450 [term 1 LEADER]: Becoming Leader. State: Replica: 90dcc6df9e944ee4ae2de3132fde0450, State: Running, Role: LEADER
I20260926 15:19:58.053830 27869 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90dcc6df9e944ee4ae2de3132fde0450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90dcc6df9e944ee4ae2de3132fde0450" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32981 } }
I20260926 15:19:58.056160 27871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90dcc6df9e944ee4ae2de3132fde0450 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90dcc6df9e944ee4ae2de3132fde0450. Latest consensus state: current_term: 1 leader_uuid: "90dcc6df9e944ee4ae2de3132fde0450" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90dcc6df9e944ee4ae2de3132fde0450" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32981 } } }
I20260926 15:19:58.056437 27871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90dcc6df9e944ee4ae2de3132fde0450 [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:58.056880 27870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90dcc6df9e944ee4ae2de3132fde0450 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90dcc6df9e944ee4ae2de3132fde0450" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90dcc6df9e944ee4ae2de3132fde0450" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32981 } } }
I20260926 15:19:58.057183 27870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90dcc6df9e944ee4ae2de3132fde0450 [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:58.057359 27882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:19:58.063102 27882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:19:58.070653 27882 catalog_manager.cc:1347] Generated new cluster ID: 78131c4c35f24e8da94914296cfa3546
I20260926 15:19:58.070742 27882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:19:58.089769 27882 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:19:58.090884 27882 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:19:58.113684 27882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90dcc6df9e944ee4ae2de3132fde0450: Generated new TSK 0
I20260926 15:19:58.114473 27882 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:19:58.130051 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 27792
W20260926 15:19:58.131001 27854 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38171
W20260926 15:19:58.131603 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 27792: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:19:58.131744 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 27792
2026-09-26T15:19:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1752 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:19:58 dist-test-slave-j6q7 krb5kdc[27890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:19:58 dist-test-slave-j6q7 krb5kdc[27890](info): set up 2 sockets
Sep 26 15:19:58 dist-test-slave-j6q7 krb5kdc[27890](info): commencing operation
krb5kdc: starting...
W20260926 15:19:59.140915 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:19:59 dist-test-slave-j6q7 krb5kdc[27890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:19:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:19:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:19:59.341750 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41987
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:19:59.642359 27903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:19:59.642587 27903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:19:59.642680 27903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:19:59.648962 27903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:19:59.649044 27903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:19:59.649088 27903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:19:59.649142 27903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:19:59.649180 27903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:19:59.658034 27903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:19:59.659024 27903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:19:59.660153 27903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:19:59.670445 27908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:59.670608 27911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:19:59.670581 27909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:19:59.671290 27903 server_base.cc:1034] running on GCE node
I20260926 15:19:59.671977 27903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:19:59.673249 27903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:19:59.674436 27903 hybrid_clock.cc:648] HybridClock initialized: now 1790435999674382 us; error 46 us; skew 500 ppm
I20260926 15:19:59.674712 27903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:19:59 dist-test-slave-j6q7 krb5kdc[27890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:19:59.685142 27903 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:19:59.686360 27903 webserver.cc:460] Webserver started at http://127.0.0.1:38685/ using document root <none> and password file <none>
I20260926 15:19:59.686908 27903 fs_manager.cc:359] Metadata directory not provided
I20260926 15:19:59.686991 27903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:19:59.687196 27903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:19:59.689745 27903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "4c2a7e5471eb4b09a826f8bc465c04af"
format_stamp: "Formatted at 2026-09-26 15:19:59 on dist-test-slave-j6q7"
server_key: "f3ab967c697ade059848e4d3ba2f2315"
server_key_iv: "4162977a6b7859b8432798733ab4b086"
server_key_version: "encryptionkey@0"
I20260926 15:19:59.690387 27903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "4c2a7e5471eb4b09a826f8bc465c04af"
format_stamp: "Formatted at 2026-09-26 15:19:59 on dist-test-slave-j6q7"
server_key: "f3ab967c697ade059848e4d3ba2f2315"
server_key_iv: "4162977a6b7859b8432798733ab4b086"
server_key_version: "encryptionkey@0"
I20260926 15:19:59.695396 27903 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 15:19:59.698335 27918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:19:59.699479 27903 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 15:19:59.699631 27903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "4c2a7e5471eb4b09a826f8bc465c04af"
format_stamp: "Formatted at 2026-09-26 15:19:59 on dist-test-slave-j6q7"
server_key: "f3ab967c697ade059848e4d3ba2f2315"
server_key_iv: "4162977a6b7859b8432798733ab4b086"
server_key_version: "encryptionkey@0"
I20260926 15:19:59.699796 27903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:19:59.705031 27903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:19:59.708220 27903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:19:59.708461 27903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:19:59.724992 27903 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41987
I20260926 15:19:59.725036 27969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41987 every 8 connection(s)
I20260926 15:19:59.726194 27903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:19:59.729457 27970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:19:59.729795 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 27903
I20260926 15:19:59.729971 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:19:59.730373 6983 external_mini_cluster.cc:1428] Setting key d981bc564350f42fb262cef99005093f
I20260926 15:19:59.738788 27973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:59.731940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60769 (local address 127.0.0.1:41987)
0926 15:19:59.732695 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:59.732709 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:59.732754 (+ 45us) server_negotiation.cc:407] Connection header received
0926 15:19:59.735509 (+ 2755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:59.735549 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:59.735946 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:59.736371 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:59.737220 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:59.737227 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:59.737241 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:59.737259 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:59.737320 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:59.737342 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:59.737517 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:59.737623 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 15:19:59.737784 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":300,"threads_started":1}
I20260926 15:19:59.741963 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:19:59.742350 27970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2a7e5471eb4b09a826f8bc465c04af: Bootstrap starting.
I20260926 15:19:59.745237 27970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c2a7e5471eb4b09a826f8bc465c04af: Neither blocks nor log segments found. Creating new log.
I20260926 15:19:59.746331 27970 log.cc:824] T 00000000000000000000000000000000 P 4c2a7e5471eb4b09a826f8bc465c04af: Log is configured to *not* fsync() on all Append() calls
I20260926 15:19:59.748220 27973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:19:59.745848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38339 (local address 127.0.0.1:41987)
0926 15:19:59.746038 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 15:19:59.746045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:19:59.746482 (+ 437us) server_negotiation.cc:407] Connection header received
0926 15:19:59.746611 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:19:59.746614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:19:59.746651 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:19:59.746718 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:19:59.747662 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:19:59.747666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:19:59.747669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:19:59.747681 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:19:59.747725 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:19:59.747737 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:19:59.747813 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 15:19:59.747940 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 15:19:59.747975 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:19:59.749325 27970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2a7e5471eb4b09a826f8bc465c04af: No bootstrap required, opened a new log
I20260926 15:19:59.756068 27970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c2a7e5471eb4b09a826f8bc465c04af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2a7e5471eb4b09a826f8bc465c04af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41987 } }
I20260926 15:19:59.756335 27970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c2a7e5471eb4b09a826f8bc465c04af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:19:59.756390 27970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c2a7e5471eb4b09a826f8bc465c04af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c2a7e5471eb4b09a826f8bc465c04af, State: Initialized, Role: FOLLOWER
I20260926 15:19:59.756893 27970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c2a7e5471eb4b09a826f8bc465c04af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2a7e5471eb4b09a826f8bc465c04af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41987 } }
I20260926 15:19:59.757091 27970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c2a7e5471eb4b09a826f8bc465c04af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:19:59.757239 27970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c2a7e5471eb4b09a826f8bc465c04af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:19:59.757387 27970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c2a7e5471eb4b09a826f8bc465c04af [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:19:59.759213 27970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c2a7e5471eb4b09a826f8bc465c04af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2a7e5471eb4b09a826f8bc465c04af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41987 } }
I20260926 15:19:59.759656 27970 leader_election.cc:302] T 00000000000000000000000000000000 P 4c2a7e5471eb4b09a826f8bc465c04af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c2a7e5471eb4b09a826f8bc465c04af; no voters:
I20260926 15:19:59.760200 27970 leader_election.cc:288] T 00000000000000000000000000000000 P 4c2a7e5471eb4b09a826f8bc465c04af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:19:59.760327 27980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c2a7e5471eb4b09a826f8bc465c04af [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:19:59.761297 27980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c2a7e5471eb4b09a826f8bc465c04af [term 1 LEADER]: Becoming Leader. State: Replica: 4c2a7e5471eb4b09a826f8bc465c04af, State: Running, Role: LEADER
I20260926 15:19:59.761785 27980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c2a7e5471eb4b09a826f8bc465c04af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2a7e5471eb4b09a826f8bc465c04af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41987 } }
I20260926 15:19:59.762409 27970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c2a7e5471eb4b09a826f8bc465c04af [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:19:59.764659 27981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2a7e5471eb4b09a826f8bc465c04af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c2a7e5471eb4b09a826f8bc465c04af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2a7e5471eb4b09a826f8bc465c04af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41987 } } }
I20260926 15:19:59.765090 27981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2a7e5471eb4b09a826f8bc465c04af [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:59.764631 27982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2a7e5471eb4b09a826f8bc465c04af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c2a7e5471eb4b09a826f8bc465c04af. Latest consensus state: current_term: 1 leader_uuid: "4c2a7e5471eb4b09a826f8bc465c04af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2a7e5471eb4b09a826f8bc465c04af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41987 } } }
I20260926 15:19:59.765465 27982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2a7e5471eb4b09a826f8bc465c04af [sys.catalog]: This master's current role is: LEADER
I20260926 15:19:59.766605 27986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:19:59.770536 27986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:19:59.778254 27986 catalog_manager.cc:1347] Generated new cluster ID: 42ae88ff5f9642e8b8fbb671fd892b76
I20260926 15:19:59.778337 27986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:19:59.804059 27986 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:19:59.804450 27986 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:19:59.812712 27986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c2a7e5471eb4b09a826f8bc465c04af: Generated new TSK 0
I20260926 15:19:59.813351 27986 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:19:59.828262 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 27903
W20260926 15:19:59.829337 27965 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60769
W20260926 15:19:59.829928 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 27903: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:19:59.830099 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 27903
2026-09-26T15:19:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1701 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:19:59 dist-test-slave-j6q7 krb5kdc[28001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:19:59 dist-test-slave-j6q7 krb5kdc[28001](info): set up 2 sockets
Sep 26 15:19:59 dist-test-slave-j6q7 krb5kdc[28001](info): commencing operation
krb5kdc: starting...
W20260926 15:20:00.844988 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:20:00 dist-test-slave-j6q7 krb5kdc[28001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:20:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:20:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:20:01.061528 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35471
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:20:01.349972 28014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:20:01.350203 28014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:20:01.350284 28014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:20:01.356748 28014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:20:01.356889 28014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:20:01.356963 28014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:20:01.357060 28014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:20:01.357124 28014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:20:01.365657 28014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:20:01.366870 28014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:20:01.368106 28014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:20:01.378456 28020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:01.378486 28022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:01.378521 28019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:20:01.379432 28014 server_base.cc:1034] running on GCE node
I20260926 15:20:01.380118 28014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:20:01.381392 28014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:20:01.382596 28014 hybrid_clock.cc:648] HybridClock initialized: now 1790436001382568 us; error 38 us; skew 500 ppm
I20260926 15:20:01.382934 28014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:20:01 dist-test-slave-j6q7 krb5kdc[28001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:20:01.385751 28014 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:20:01.386961 28014 webserver.cc:460] Webserver started at http://127.0.0.1:40549/ using document root <none> and password file <none>
I20260926 15:20:01.387528 28014 fs_manager.cc:359] Metadata directory not provided
I20260926 15:20:01.387650 28014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:20:01.387902 28014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:20:01.390437 28014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "dee82ccc90564900b1293c2575570ff6"
format_stamp: "Formatted at 2026-09-26 15:20:01 on dist-test-slave-j6q7"
server_key: "73f6ab82302565acf3f1dadb7c1a7094"
server_key_iv: "9c00c8b2ac1e387a0112345a3405326a"
server_key_version: "encryptionkey@0"
I20260926 15:20:01.391114 28014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "dee82ccc90564900b1293c2575570ff6"
format_stamp: "Formatted at 2026-09-26 15:20:01 on dist-test-slave-j6q7"
server_key: "73f6ab82302565acf3f1dadb7c1a7094"
server_key_iv: "9c00c8b2ac1e387a0112345a3405326a"
server_key_version: "encryptionkey@0"
I20260926 15:20:01.395522 28014 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 15:20:01.398406 28029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:20:01.399462 28014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 15:20:01.399614 28014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "dee82ccc90564900b1293c2575570ff6"
format_stamp: "Formatted at 2026-09-26 15:20:01 on dist-test-slave-j6q7"
server_key: "73f6ab82302565acf3f1dadb7c1a7094"
server_key_iv: "9c00c8b2ac1e387a0112345a3405326a"
server_key_version: "encryptionkey@0"
I20260926 15:20:01.399773 28014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:20:01.433264 28014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:20:01.437233 28014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:20:01.437492 28014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:20:01.454166 28014 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35471
I20260926 15:20:01.454203 28080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35471 every 8 connection(s)
I20260926 15:20:01.455407 28014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:20:01.458729 28081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:20:01.459266 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 28014
I20260926 15:20:01.459427 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:20:01.459935 6983 external_mini_cluster.cc:1428] Setting key 59dc81a81a0f4f86d9dbf0f156305abe
I20260926 15:20:01.471088 28081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee82ccc90564900b1293c2575570ff6: Bootstrap starting.
Sep 26 15:20:01 dist-test-slave-j6q7 krb5kdc[28001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 15:20:01.474398 28081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dee82ccc90564900b1293c2575570ff6: Neither blocks nor log segments found. Creating new log.
I20260926 15:20:01.475692 28081 log.cc:824] T 00000000000000000000000000000000 P dee82ccc90564900b1293c2575570ff6: Log is configured to *not* fsync() on all Append() calls
I20260926 15:20:01.478432 28081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee82ccc90564900b1293c2575570ff6: No bootstrap required, opened a new log
I20260926 15:20:01.481395 28084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:01.461677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.249:33665 (local address 127.0.0.1:35471)
0926 15:20:01.462343 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:01.462353 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:01.462386 (+ 33us) server_negotiation.cc:407] Connection header received
0926 15:20:01.464179 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:01.464202 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:01.464639 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:01.465030 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:01.466477 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:01.467667 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:01.469246 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:01.469765 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:01.469887 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:01.473988 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:01.474001 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:01.474021 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:01.474060 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:01.477206 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:01.477994 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:01.478000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:01.478005 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:01.478150 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:01.478647 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:01.478652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:01.478656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:01.479252 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:01.479521 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:01.479850 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 15:20:01.480178 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":298,"threads_started":1}
I20260926 15:20:01.485124 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:20:01.487176 28081 raft_consensus.cc:348] T 00000000000000000000000000000000 P dee82ccc90564900b1293c2575570ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee82ccc90564900b1293c2575570ff6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35471 } }
I20260926 15:20:01.487628 28081 raft_consensus.cc:374] T 00000000000000000000000000000000 P dee82ccc90564900b1293c2575570ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:20:01.487751 28081 raft_consensus.cc:729] T 00000000000000000000000000000000 P dee82ccc90564900b1293c2575570ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee82ccc90564900b1293c2575570ff6, State: Initialized, Role: FOLLOWER
I20260926 15:20:01.488389 28081 consensus_queue.cc:260] T 00000000000000000000000000000000 P dee82ccc90564900b1293c2575570ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee82ccc90564900b1293c2575570ff6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35471 } }
I20260926 15:20:01.488638 28081 raft_consensus.cc:388] T 00000000000000000000000000000000 P dee82ccc90564900b1293c2575570ff6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:20:01.488772 28081 raft_consensus.cc:482] T 00000000000000000000000000000000 P dee82ccc90564900b1293c2575570ff6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:20:01.488996 28081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dee82ccc90564900b1293c2575570ff6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:20:01.491120 28081 raft_consensus.cc:504] T 00000000000000000000000000000000 P dee82ccc90564900b1293c2575570ff6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee82ccc90564900b1293c2575570ff6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35471 } }
I20260926 15:20:01.491603 28081 leader_election.cc:302] T 00000000000000000000000000000000 P dee82ccc90564900b1293c2575570ff6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dee82ccc90564900b1293c2575570ff6; no voters:
I20260926 15:20:01.492249 28081 leader_election.cc:288] T 00000000000000000000000000000000 P dee82ccc90564900b1293c2575570ff6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:20:01.492329 28091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dee82ccc90564900b1293c2575570ff6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:20:01.494558 28081 sys_catalog.cc:564] T 00000000000000000000000000000000 P dee82ccc90564900b1293c2575570ff6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:20:01.498907 28091 raft_consensus.cc:686] T 00000000000000000000000000000000 P dee82ccc90564900b1293c2575570ff6 [term 1 LEADER]: Becoming Leader. State: Replica: dee82ccc90564900b1293c2575570ff6, State: Running, Role: LEADER
I20260926 15:20:01.499511 28091 consensus_queue.cc:237] T 00000000000000000000000000000000 P dee82ccc90564900b1293c2575570ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee82ccc90564900b1293c2575570ff6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35471 } }
I20260926 15:20:01.502300 28084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:01.489186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.249:55933 (local address 127.0.0.1:35471)
0926 15:20:01.489365 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:01.489372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:01.489702 (+ 330us) server_negotiation.cc:407] Connection header received
0926 15:20:01.489880 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:01.489884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:01.489968 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:01.490072 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:01.491303 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:01.492077 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:01.493563 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:01.494020 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:01.494155 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:01.496673 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:01.496682 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:01.496689 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:01.496734 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:01.499411 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:01.500188 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:01.500200 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:01.500203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:01.500306 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:01.500739 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:01.500744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:01.500749 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:01.501225 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:01.501461 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:01.501806 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 15:20:01.501971 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 15:20:01.506628 28092 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee82ccc90564900b1293c2575570ff6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dee82ccc90564900b1293c2575570ff6. Latest consensus state: current_term: 1 leader_uuid: "dee82ccc90564900b1293c2575570ff6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee82ccc90564900b1293c2575570ff6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35471 } } }
I20260926 15:20:01.507411 28092 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee82ccc90564900b1293c2575570ff6 [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:01.507297 28093 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee82ccc90564900b1293c2575570ff6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dee82ccc90564900b1293c2575570ff6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee82ccc90564900b1293c2575570ff6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35471 } } }
I20260926 15:20:01.507602 28093 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee82ccc90564900b1293c2575570ff6 [sys.catalog]: This master's current role is: LEADER
W20260926 15:20:01.511081 28105 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dee82ccc90564900b1293c2575570ff6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:20:01.511221 28105 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:20:01.513612 28106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:20:01.515244 28106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:20:01.522806 28106 catalog_manager.cc:1347] Generated new cluster ID: 2191f5b356b54287bbc4e362f65a8801
I20260926 15:20:01.522915 28106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:20:01.555958 28106 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:20:01.557308 28106 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:20:01.563491 28106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dee82ccc90564900b1293c2575570ff6: Generated new TSK 0
I20260926 15:20:01.564473 28106 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:20:01.580591 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 28014
I20260926 15:20:01.581707 28076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:20:01.675881 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 28014
W20260926 15:20:01.689819 27999 connection.cc:535] client connection to 127.0.0.1:35471 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35471): Connection reset by peer (error 104)
2026-09-26T15:20:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1843 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:20:01 dist-test-slave-j6q7 krb5kdc[28113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:20:01 dist-test-slave-j6q7 krb5kdc[28113](info): set up 2 sockets
Sep 26 15:20:01 dist-test-slave-j6q7 krb5kdc[28113](info): commencing operation
krb5kdc: starting...
W20260926 15:20:02.764870 6983 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:20:02 dist-test-slave-j6q7 krb5kdc[28113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:20:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:20:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:20:02.983886 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42479
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:20:03.298964 28126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:20:03.299223 28126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:20:03.299309 28126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:20:03.306228 28126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:20:03.306329 28126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:20:03.306376 28126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:20:03.306442 28126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:20:03.306494 28126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:20:03.315244 28126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:20:03.316298 28126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:20:03.317476 28126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:20:03.328971 28134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:03.328907 28131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:03.328936 28132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:20:03.329608 28126 server_base.cc:1034] running on GCE node
I20260926 15:20:03.330190 28126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:20:03.331256 28126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:20:03.332432 28126 hybrid_clock.cc:648] HybridClock initialized: now 1790436003332388 us; error 46 us; skew 500 ppm
I20260926 15:20:03.332731 28126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:20:03 dist-test-slave-j6q7 krb5kdc[28113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:20:03.335594 28126 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:20:03.336799 28126 webserver.cc:460] Webserver started at http://127.0.0.1:33285/ using document root <none> and password file <none>
I20260926 15:20:03.337410 28126 fs_manager.cc:359] Metadata directory not provided
I20260926 15:20:03.337522 28126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:20:03.337805 28126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:20:03.340611 28126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "6861a288f2c245328d55b7fa521d35ac"
format_stamp: "Formatted at 2026-09-26 15:20:03 on dist-test-slave-j6q7"
server_key: "33bc56cbe4479fe5a96542d725f1c330"
server_key_iv: "6216a2156586a449bdabaa494bca7046"
server_key_version: "encryptionkey@0"
I20260926 15:20:03.341347 28126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "6861a288f2c245328d55b7fa521d35ac"
format_stamp: "Formatted at 2026-09-26 15:20:03 on dist-test-slave-j6q7"
server_key: "33bc56cbe4479fe5a96542d725f1c330"
server_key_iv: "6216a2156586a449bdabaa494bca7046"
server_key_version: "encryptionkey@0"
I20260926 15:20:03.345929 28126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 15:20:03.348923 28141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:20:03.349973 28126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:20:03.350124 28126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "6861a288f2c245328d55b7fa521d35ac"
format_stamp: "Formatted at 2026-09-26 15:20:03 on dist-test-slave-j6q7"
server_key: "33bc56cbe4479fe5a96542d725f1c330"
server_key_iv: "6216a2156586a449bdabaa494bca7046"
server_key_version: "encryptionkey@0"
I20260926 15:20:03.350284 28126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:20:03.364216 28126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:20:03.367568 28126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:20:03.367807 28126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:20:03.384289 28126 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42479
I20260926 15:20:03.384363 28192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42479 every 8 connection(s)
I20260926 15:20:03.385509 28126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:20:03.389010 28193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:20:03.394552 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 28126
I20260926 15:20:03.394733 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:20:03.395290 6983 external_mini_cluster.cc:1428] Setting key 19967ce1ce6db5cf834f68fd0fdbe91a
I20260926 15:20:03.398510 28193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6861a288f2c245328d55b7fa521d35ac: Bootstrap starting.
I20260926 15:20:03.401543 28193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6861a288f2c245328d55b7fa521d35ac: Neither blocks nor log segments found. Creating new log.
I20260926 15:20:03.402637 28193 log.cc:824] T 00000000000000000000000000000000 P 6861a288f2c245328d55b7fa521d35ac: Log is configured to *not* fsync() on all Append() calls
I20260926 15:20:03.405591 28193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6861a288f2c245328d55b7fa521d35ac: No bootstrap required, opened a new log
I20260926 15:20:03.407413 28196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:03.396913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.249:55891 (local address 127.0.0.1:42479)
0926 15:20:03.397549 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:03.397559 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:03.397592 (+ 33us) server_negotiation.cc:407] Connection header received
0926 15:20:03.399050 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:03.399069 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:03.399383 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:03.399657 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:03.400253 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:03.401477 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:03.403276 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:03.404112 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:03.404229 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:03.405448 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:03.405455 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:03.405484 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:20:03.405998 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:03.406056 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:20:03.406082 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:03.406179 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:03.406444 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 15:20:03.406602 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":292,"threads_started":1}
I20260926 15:20:03.410996 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:20:03.413400 28193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6861a288f2c245328d55b7fa521d35ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6861a288f2c245328d55b7fa521d35ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42479 } }
I20260926 15:20:03.413815 28193 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6861a288f2c245328d55b7fa521d35ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:20:03.413926 28193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6861a288f2c245328d55b7fa521d35ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6861a288f2c245328d55b7fa521d35ac, State: Initialized, Role: FOLLOWER
I20260926 15:20:03.414523 28193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6861a288f2c245328d55b7fa521d35ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6861a288f2c245328d55b7fa521d35ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42479 } }
I20260926 15:20:03.414783 28193 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6861a288f2c245328d55b7fa521d35ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:20:03.414938 28193 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6861a288f2c245328d55b7fa521d35ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:20:03.415171 28193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6861a288f2c245328d55b7fa521d35ac [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:20:03.417183 28193 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6861a288f2c245328d55b7fa521d35ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6861a288f2c245328d55b7fa521d35ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42479 } }
I20260926 15:20:03.417619 28193 leader_election.cc:302] T 00000000000000000000000000000000 P 6861a288f2c245328d55b7fa521d35ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6861a288f2c245328d55b7fa521d35ac; no voters:
I20260926 15:20:03.418232 28193 leader_election.cc:288] T 00000000000000000000000000000000 P 6861a288f2c245328d55b7fa521d35ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:20:03.418313 28203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6861a288f2c245328d55b7fa521d35ac [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:20:03.420409 28193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6861a288f2c245328d55b7fa521d35ac [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:20:03.421126 28196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:03.415057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.249:48771 (local address 127.0.0.1:42479)
0926 15:20:03.415224 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:03.415229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:03.415676 (+ 447us) server_negotiation.cc:407] Connection header received
0926 15:20:03.415850 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:03.415854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:03.415917 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:03.416012 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:03.416394 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:03.417090 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:03.418435 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:03.419087 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:03.419726 (+ 639us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:03.420305 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:03.420310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:03.420313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:20:03.420335 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:03.420387 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:20:03.420405 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:03.420521 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:03.420710 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 15:20:03.420757 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 15:20:03.425406 28203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6861a288f2c245328d55b7fa521d35ac [term 1 LEADER]: Becoming Leader. State: Replica: 6861a288f2c245328d55b7fa521d35ac, State: Running, Role: LEADER
I20260926 15:20:03.425976 28203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6861a288f2c245328d55b7fa521d35ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6861a288f2c245328d55b7fa521d35ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42479 } }
I20260926 15:20:03.429118 28204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6861a288f2c245328d55b7fa521d35ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6861a288f2c245328d55b7fa521d35ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6861a288f2c245328d55b7fa521d35ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42479 } } }
I20260926 15:20:03.429152 28205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6861a288f2c245328d55b7fa521d35ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6861a288f2c245328d55b7fa521d35ac. Latest consensus state: current_term: 1 leader_uuid: "6861a288f2c245328d55b7fa521d35ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6861a288f2c245328d55b7fa521d35ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42479 } } }
I20260926 15:20:03.429476 28204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6861a288f2c245328d55b7fa521d35ac [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:03.429513 28205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6861a288f2c245328d55b7fa521d35ac [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:03.430871 28216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:20:03.434723 28216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:20:03.442201 28216 catalog_manager.cc:1347] Generated new cluster ID: 98d9c09da0f34f94a97c23c79dee356e
I20260926 15:20:03.442348 28216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:20:03.454926 28216 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:20:03.456489 28216 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:20:03.470965 28216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6861a288f2c245328d55b7fa521d35ac: Generated new TSK 0
I20260926 15:20:03.471758 28216 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:20:03.479624 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 28126
W20260926 15:20:03.480634 28188 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.249:55891
W20260926 15:20:03.481428 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 28126: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:20:03.481560 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 28126
W20260926 15:20:03.497351 28111 connection.cc:535] client connection to 127.0.0.1:42479 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42479): Connection reset by peer (error 104)
2026-09-26T15:20:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1807 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:20:03 dist-test-slave-j6q7 krb5kdc[28224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:20:03 dist-test-slave-j6q7 krb5kdc[28224](info): set up 2 sockets
Sep 26 15:20:03 dist-test-slave-j6q7 krb5kdc[28224](info): commencing operation
krb5kdc: starting...
W20260926 15:20:04.501209 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:20:04 dist-test-slave-j6q7 krb5kdc[28224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:20:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:20:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:20:04.739858 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45953
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:20:05.054514 28237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:20:05.054774 28237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:20:05.054961 28237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:20:05.062142 28237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:20:05.062263 28237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:20:05.062350 28237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:20:05.062453 28237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:20:05.062528 28237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:20:05.072131 28237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:20:05.073393 28237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:20:05.074601 28237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:20:05.086109 28243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:05.086054 28245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:05.086123 28242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:20:05.086902 28237 server_base.cc:1034] running on GCE node
I20260926 15:20:05.087634 28237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:20:05.088972 28237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:20:05.090199 28237 hybrid_clock.cc:648] HybridClock initialized: now 1790436005090180 us; error 43 us; skew 500 ppm
I20260926 15:20:05.090523 28237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:20:05 dist-test-slave-j6q7 krb5kdc[28224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:20:05.099053 28237 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:20:05.100499 28237 webserver.cc:460] Webserver started at http://127.0.0.1:37581/ using document root <none> and password file <none>
I20260926 15:20:05.101219 28237 fs_manager.cc:359] Metadata directory not provided
I20260926 15:20:05.101362 28237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:20:05.101651 28237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:20:05.104442 28237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "a89f545204f945fc92de7446f3832199"
format_stamp: "Formatted at 2026-09-26 15:20:05 on dist-test-slave-j6q7"
server_key: "d15f128e1d77978d8dd3c4b76dba3d87"
server_key_iv: "cad2b9c3bfb90654a3b787b2c67fb170"
server_key_version: "encryptionkey@0"
I20260926 15:20:05.105212 28237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "a89f545204f945fc92de7446f3832199"
format_stamp: "Formatted at 2026-09-26 15:20:05 on dist-test-slave-j6q7"
server_key: "d15f128e1d77978d8dd3c4b76dba3d87"
server_key_iv: "cad2b9c3bfb90654a3b787b2c67fb170"
server_key_version: "encryptionkey@0"
I20260926 15:20:05.109978 28237 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 15:20:05.113229 28252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:20:05.114461 28237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:20:05.114638 28237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "a89f545204f945fc92de7446f3832199"
format_stamp: "Formatted at 2026-09-26 15:20:05 on dist-test-slave-j6q7"
server_key: "d15f128e1d77978d8dd3c4b76dba3d87"
server_key_iv: "cad2b9c3bfb90654a3b787b2c67fb170"
server_key_version: "encryptionkey@0"
I20260926 15:20:05.114845 28237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:20:05.120253 28237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:20:05.124260 28237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:20:05.124552 28237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:20:05.142149 28237 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45953
I20260926 15:20:05.142282 28303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45953 every 8 connection(s)
I20260926 15:20:05.143410 28237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:20:05.146868 28304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:20:05.152632 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 28237
I20260926 15:20:05.152843 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:20:05.153367 6983 external_mini_cluster.cc:1428] Setting key fb7538a4375dbda7a7f9ee9d479017ad
I20260926 15:20:05.157445 28304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a89f545204f945fc92de7446f3832199: Bootstrap starting.
I20260926 15:20:05.160527 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:05.155030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.249:56969 (local address 127.0.0.1:45953)
0926 15:20:05.155835 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:05.155845 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:05.155881 (+ 36us) server_negotiation.cc:407] Connection header received
0926 15:20:05.157425 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:05.157448 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:05.157812 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:05.158105 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:05.159103 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:05.159109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:05.159120 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:20:05.159132 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:05.159184 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:20:05.159205 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:05.159367 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:05.159503 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 15:20:05.159693 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":398,"threads_started":1}
I20260926 15:20:05.160864 28304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a89f545204f945fc92de7446f3832199: Neither blocks nor log segments found. Creating new log.
I20260926 15:20:05.162062 28304 log.cc:824] T 00000000000000000000000000000000 P a89f545204f945fc92de7446f3832199: Log is configured to *not* fsync() on all Append() calls
I20260926 15:20:05.164093 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:20:05.165012 28304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a89f545204f945fc92de7446f3832199: No bootstrap required, opened a new log
I20260926 15:20:05.171325 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:05.168950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.249:33879 (local address 127.0.0.1:45953)
0926 15:20:05.169138 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:05.169144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:05.169616 (+ 472us) server_negotiation.cc:407] Connection header received
0926 15:20:05.169715 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:05.169721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:05.169777 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:05.169877 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:05.170702 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:05.170708 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:05.170713 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:20:05.170729 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:05.170775 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:20:05.170795 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:05.170906 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:05.170976 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 15:20:05.171036 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 15:20:05.174247 28304 raft_consensus.cc:348] T 00000000000000000000000000000000 P a89f545204f945fc92de7446f3832199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89f545204f945fc92de7446f3832199" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45953 } }
I20260926 15:20:05.174544 28304 raft_consensus.cc:374] T 00000000000000000000000000000000 P a89f545204f945fc92de7446f3832199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:20:05.174613 28304 raft_consensus.cc:729] T 00000000000000000000000000000000 P a89f545204f945fc92de7446f3832199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a89f545204f945fc92de7446f3832199, State: Initialized, Role: FOLLOWER
I20260926 15:20:05.175115 28304 consensus_queue.cc:260] T 00000000000000000000000000000000 P a89f545204f945fc92de7446f3832199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89f545204f945fc92de7446f3832199" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45953 } }
I20260926 15:20:05.175318 28304 raft_consensus.cc:388] T 00000000000000000000000000000000 P a89f545204f945fc92de7446f3832199 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:20:05.175424 28304 raft_consensus.cc:482] T 00000000000000000000000000000000 P a89f545204f945fc92de7446f3832199 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:20:05.175557 28304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a89f545204f945fc92de7446f3832199 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:20:05.177747 28304 raft_consensus.cc:504] T 00000000000000000000000000000000 P a89f545204f945fc92de7446f3832199 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89f545204f945fc92de7446f3832199" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45953 } }
I20260926 15:20:05.178107 28304 leader_election.cc:302] T 00000000000000000000000000000000 P a89f545204f945fc92de7446f3832199 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a89f545204f945fc92de7446f3832199; no voters:
I20260926 15:20:05.178565 28304 leader_election.cc:288] T 00000000000000000000000000000000 P a89f545204f945fc92de7446f3832199 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:20:05.178653 28314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a89f545204f945fc92de7446f3832199 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:20:05.179351 28314 raft_consensus.cc:686] T 00000000000000000000000000000000 P a89f545204f945fc92de7446f3832199 [term 1 LEADER]: Becoming Leader. State: Replica: a89f545204f945fc92de7446f3832199, State: Running, Role: LEADER
I20260926 15:20:05.179834 28314 consensus_queue.cc:237] T 00000000000000000000000000000000 P a89f545204f945fc92de7446f3832199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89f545204f945fc92de7446f3832199" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45953 } }
I20260926 15:20:05.180447 28304 sys_catalog.cc:564] T 00000000000000000000000000000000 P a89f545204f945fc92de7446f3832199 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:20:05.182787 28315 sys_catalog.cc:455] T 00000000000000000000000000000000 P a89f545204f945fc92de7446f3832199 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a89f545204f945fc92de7446f3832199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89f545204f945fc92de7446f3832199" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45953 } } }
I20260926 15:20:05.183239 28315 sys_catalog.cc:458] T 00000000000000000000000000000000 P a89f545204f945fc92de7446f3832199 [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:05.183938 28316 sys_catalog.cc:455] T 00000000000000000000000000000000 P a89f545204f945fc92de7446f3832199 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a89f545204f945fc92de7446f3832199. Latest consensus state: current_term: 1 leader_uuid: "a89f545204f945fc92de7446f3832199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89f545204f945fc92de7446f3832199" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45953 } } }
I20260926 15:20:05.184353 28316 sys_catalog.cc:458] T 00000000000000000000000000000000 P a89f545204f945fc92de7446f3832199 [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:05.184650 28324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:20:05.189368 28324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:20:05.196727 28324 catalog_manager.cc:1347] Generated new cluster ID: 3488ce89c9e842dea4fafd64cfb03075
I20260926 15:20:05.196836 28324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:20:05.209600 28324 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:20:05.209935 28324 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:20:05.218132 28324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a89f545204f945fc92de7446f3832199: Generated new TSK 0
I20260926 15:20:05.218726 28324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:20:05.227159 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 28237
W20260926 15:20:05.227964 28299 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.249:56969
W20260926 15:20:05.228585 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 28237: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:20:05.228734 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 28237
2026-09-26T15:20:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1746 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:20:05 dist-test-slave-j6q7 krb5kdc[28335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:20:05 dist-test-slave-j6q7 krb5kdc[28335](info): set up 2 sockets
Sep 26 15:20:05 dist-test-slave-j6q7 krb5kdc[28335](info): commencing operation
krb5kdc: starting...
W20260926 15:20:06.236984 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:20:06 dist-test-slave-j6q7 krb5kdc[28335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:20:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:20:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:20:06.429309 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44947
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:20:06.715970 28348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:20:06.716224 28348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:20:06.716379 28348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:20:06.722361 28348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:20:06.722477 28348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:20:06.722568 28348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:20:06.722661 28348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:20:06.722733 28348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:20:06.731204 28348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44947
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:20:06.732209 28348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:20:06.733332 28348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:20:06.743459 28353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:06.743517 28354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:06.743595 28356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:20:06.744051 28348 server_base.cc:1034] running on GCE node
I20260926 15:20:06.744756 28348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:20:06.745959 28348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:20:06.747131 28348 hybrid_clock.cc:648] HybridClock initialized: now 1790436006747102 us; error 23 us; skew 500 ppm
I20260926 15:20:06.747429 28348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:20:06 dist-test-slave-j6q7 krb5kdc[28335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:20:06.755538 28348 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:20:06.756686 28348 webserver.cc:460] Webserver started at http://127.0.0.1:34693/ using document root <none> and password file <none>
I20260926 15:20:06.757328 28348 fs_manager.cc:359] Metadata directory not provided
I20260926 15:20:06.757437 28348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:20:06.757648 28348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:20:06.760144 28348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "2bf9bfc388114d6ab852a61f6897449f"
format_stamp: "Formatted at 2026-09-26 15:20:06 on dist-test-slave-j6q7"
server_key: "844a43d1c14cd51cb77c171cc0c44329"
server_key_iv: "0b90dd736261796ca76cae4145783efd"
server_key_version: "encryptionkey@0"
I20260926 15:20:06.760797 28348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "2bf9bfc388114d6ab852a61f6897449f"
format_stamp: "Formatted at 2026-09-26 15:20:06 on dist-test-slave-j6q7"
server_key: "844a43d1c14cd51cb77c171cc0c44329"
server_key_iv: "0b90dd736261796ca76cae4145783efd"
server_key_version: "encryptionkey@0"
I20260926 15:20:06.765148 28348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:20:06.768169 28363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:20:06.769351 28348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:20:06.769513 28348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "2bf9bfc388114d6ab852a61f6897449f"
format_stamp: "Formatted at 2026-09-26 15:20:06 on dist-test-slave-j6q7"
server_key: "844a43d1c14cd51cb77c171cc0c44329"
server_key_iv: "0b90dd736261796ca76cae4145783efd"
server_key_version: "encryptionkey@0"
I20260926 15:20:06.769680 28348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:20:06.784693 28348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:20:06.788040 28348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:20:06.788268 28348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:20:06.803992 28348 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44947
I20260926 15:20:06.804023 28414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44947 every 8 connection(s)
I20260926 15:20:06.805184 28348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:20:06.806412 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 28348
I20260926 15:20:06.806559 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:20:06.807006 6983 external_mini_cluster.cc:1428] Setting key ae6069fbeb66ff369d563d36eaee6903
I20260926 15:20:06.809126 28415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 15:20:06 dist-test-slave-j6q7 krb5kdc[28335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 15:20:06.820286 28415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bf9bfc388114d6ab852a61f6897449f: Bootstrap starting.
I20260926 15:20:06.822933 28415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bf9bfc388114d6ab852a61f6897449f: Neither blocks nor log segments found. Creating new log.
I20260926 15:20:06.823767 28415 log.cc:824] T 00000000000000000000000000000000 P 2bf9bfc388114d6ab852a61f6897449f: Log is configured to *not* fsync() on all Append() calls
I20260926 15:20:06.826269 28415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bf9bfc388114d6ab852a61f6897449f: No bootstrap required, opened a new log
I20260926 15:20:06.826889 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:06.808452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.249:42439 (local address 127.0.0.1:44947)
0926 15:20:06.809134 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:06.809144 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:06.809204 (+ 60us) server_negotiation.cc:407] Connection header received
0926 15:20:06.810638 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:06.810655 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:06.810970 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:06.811236 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:06.812723 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:06.813910 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:06.815727 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:06.816263 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:06.816384 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:06.820529 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:06.820548 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:06.820559 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:06.820592 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:06.823211 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:06.823848 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:06.823854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:06.823858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:06.824025 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:06.824464 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:06.824469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:06.824473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:06.825002 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:06.825291 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:06.825754 (+ 463us) server_negotiation.cc:299] Negotiation successful
0926 15:20:06.826036 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":329,"threads_started":1}
I20260926 15:20:06.830318 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:20:06.833774 28415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bf9bfc388114d6ab852a61f6897449f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf9bfc388114d6ab852a61f6897449f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44947 } }
I20260926 15:20:06.834134 28415 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bf9bfc388114d6ab852a61f6897449f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:20:06.834244 28415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bf9bfc388114d6ab852a61f6897449f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf9bfc388114d6ab852a61f6897449f, State: Initialized, Role: FOLLOWER
I20260926 15:20:06.834884 28415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bf9bfc388114d6ab852a61f6897449f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf9bfc388114d6ab852a61f6897449f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44947 } }
I20260926 15:20:06.835158 28415 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bf9bfc388114d6ab852a61f6897449f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:20:06.835311 28415 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bf9bfc388114d6ab852a61f6897449f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:20:06.835485 28415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bf9bfc388114d6ab852a61f6897449f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:20:06.837849 28415 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bf9bfc388114d6ab852a61f6897449f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf9bfc388114d6ab852a61f6897449f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44947 } }
I20260926 15:20:06.838270 28415 leader_election.cc:302] T 00000000000000000000000000000000 P 2bf9bfc388114d6ab852a61f6897449f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bf9bfc388114d6ab852a61f6897449f; no voters:
I20260926 15:20:06.838851 28415 leader_election.cc:288] T 00000000000000000000000000000000 P 2bf9bfc388114d6ab852a61f6897449f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:20:06.838965 28425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bf9bfc388114d6ab852a61f6897449f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:20:06.840921 28415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bf9bfc388114d6ab852a61f6897449f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:20:06.845414 28425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bf9bfc388114d6ab852a61f6897449f [term 1 LEADER]: Becoming Leader. State: Replica: 2bf9bfc388114d6ab852a61f6897449f, State: Running, Role: LEADER
I20260926 15:20:06.845952 28425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bf9bfc388114d6ab852a61f6897449f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf9bfc388114d6ab852a61f6897449f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44947 } }
I20260926 15:20:06.848225 28426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bf9bfc388114d6ab852a61f6897449f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bf9bfc388114d6ab852a61f6897449f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf9bfc388114d6ab852a61f6897449f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44947 } } }
I20260926 15:20:06.848212 28427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bf9bfc388114d6ab852a61f6897449f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bf9bfc388114d6ab852a61f6897449f. Latest consensus state: current_term: 1 leader_uuid: "2bf9bfc388114d6ab852a61f6897449f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf9bfc388114d6ab852a61f6897449f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44947 } } }
I20260926 15:20:06.848786 28426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bf9bfc388114d6ab852a61f6897449f [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:06.848799 28427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bf9bfc388114d6ab852a61f6897449f [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:06.852028 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:06.834134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.249:49345 (local address 127.0.0.1:44947)
0926 15:20:06.834280 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:06.834284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:06.834761 (+ 477us) server_negotiation.cc:407] Connection header received
0926 15:20:06.834880 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:06.834885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:06.834955 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:06.835054 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:06.836441 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:06.837468 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:06.838769 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:06.839626 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:06.839731 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:06.842944 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:06.842953 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:06.842959 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:06.843000 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:06.847254 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:06.848411 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:06.848419 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:06.848425 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:06.849486 (+ 1061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:06.850063 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:06.850068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:06.850073 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:06.850413 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:06.850661 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:06.851548 (+ 887us) server_negotiation.cc:299] Negotiation successful
0926 15:20:06.851706 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 15:20:06.857227 28436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:20:06.862112 28436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:20:06.868868 28436 catalog_manager.cc:1347] Generated new cluster ID: 75d3c57fef634628b06ec60b7e8e11b2
I20260926 15:20:06.868963 28436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:20:06.890355 28436 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:20:06.891614 28436 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:20:06.902770 28436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bf9bfc388114d6ab852a61f6897449f: Generated new TSK 0
I20260926 15:20:06.903455 28436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:20:06.914798 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 28348
I20260926 15:20:06.916028 28410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:20:07.022238 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 28348
W20260926 15:20:07.038034 28333 connection.cc:535] client connection to 127.0.0.1:44947 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44947): Connection reset by peer (error 104)
2026-09-26T15:20:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1793 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:20:07 dist-test-slave-j6q7 krb5kdc[28447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:20:07 dist-test-slave-j6q7 krb5kdc[28447](info): set up 2 sockets
Sep 26 15:20:07 dist-test-slave-j6q7 krb5kdc[28447](info): commencing operation
krb5kdc: starting...
W20260926 15:20:08.024670 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:20:08 dist-test-slave-j6q7 krb5kdc[28447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:20:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:20:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:20:08.224050 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37557
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:20:08.511503 28460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:20:08.511775 28460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:20:08.511919 28460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:20:08.518110 28460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:20:08.518256 28460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:20:08.518347 28460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:20:08.518445 28460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:20:08.518519 28460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:20:08.526883 28460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37557
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:20:08.527912 28460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:20:08.529038 28460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:20:08.539666 28468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:08.539626 28465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:08.539675 28466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:20:08.540587 28460 server_base.cc:1034] running on GCE node
I20260926 15:20:08.541306 28460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:20:08.542374 28460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:20:08.543596 28460 hybrid_clock.cc:648] HybridClock initialized: now 1790436008543562 us; error 46 us; skew 500 ppm
I20260926 15:20:08.543892 28460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:20:08 dist-test-slave-j6q7 krb5kdc[28447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:20:08.546646 28460 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:20:08.547855 28460 webserver.cc:460] Webserver started at http://127.0.0.1:45251/ using document root <none> and password file <none>
I20260926 15:20:08.548414 28460 fs_manager.cc:359] Metadata directory not provided
I20260926 15:20:08.548540 28460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:20:08.548791 28460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:20:08.551226 28460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "894ed48ecdad45b0b2974f39d0216e18"
format_stamp: "Formatted at 2026-09-26 15:20:08 on dist-test-slave-j6q7"
server_key: "90cd710fac2d8f6b8fa1522bc81bcd5a"
server_key_iv: "c7f38b96935fcbf2947023812ad9fefe"
server_key_version: "encryptionkey@0"
I20260926 15:20:08.551888 28460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "894ed48ecdad45b0b2974f39d0216e18"
format_stamp: "Formatted at 2026-09-26 15:20:08 on dist-test-slave-j6q7"
server_key: "90cd710fac2d8f6b8fa1522bc81bcd5a"
server_key_iv: "c7f38b96935fcbf2947023812ad9fefe"
server_key_version: "encryptionkey@0"
I20260926 15:20:08.556768 28460 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 15:20:08.560021 28475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:20:08.561344 28460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:20:08.561512 28460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "894ed48ecdad45b0b2974f39d0216e18"
format_stamp: "Formatted at 2026-09-26 15:20:08 on dist-test-slave-j6q7"
server_key: "90cd710fac2d8f6b8fa1522bc81bcd5a"
server_key_iv: "c7f38b96935fcbf2947023812ad9fefe"
server_key_version: "encryptionkey@0"
I20260926 15:20:08.561695 28460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:20:08.580018 28460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:20:08.583575 28460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:20:08.583820 28460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:20:08.602885 28460 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37557
I20260926 15:20:08.602931 28526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37557 every 8 connection(s)
I20260926 15:20:08.604243 28460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:20:08.607650 28527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:20:08.610692 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 28460
I20260926 15:20:08.610911 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:20:08.611505 6983 external_mini_cluster.cc:1428] Setting key bae75b258607a541a58b7801e231e770
I20260926 15:20:08.618568 28527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 894ed48ecdad45b0b2974f39d0216e18: Bootstrap starting.
I20260926 15:20:08.621599 28527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 894ed48ecdad45b0b2974f39d0216e18: Neither blocks nor log segments found. Creating new log.
I20260926 15:20:08.622557 28530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:08.613198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.249:46759 (local address 127.0.0.1:37557)
0926 15:20:08.613819 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:08.613829 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:08.613863 (+ 34us) server_negotiation.cc:407] Connection header received
0926 15:20:08.615382 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:08.615402 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:08.615768 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:08.616089 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:08.616538 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:08.617687 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:08.618798 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:08.619340 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:08.619494 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:08.620650 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:08.620657 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:08.620669 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:20:08.621188 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:08.621269 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:20:08.621289 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:08.621423 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:08.621648 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 15:20:08.621800 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":266,"threads_started":1}
I20260926 15:20:08.622656 28527 log.cc:824] T 00000000000000000000000000000000 P 894ed48ecdad45b0b2974f39d0216e18: Log is configured to *not* fsync() on all Append() calls
I20260926 15:20:08.625344 28527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 894ed48ecdad45b0b2974f39d0216e18: No bootstrap required, opened a new log
I20260926 15:20:08.625627 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:20:08.633304 28527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 894ed48ecdad45b0b2974f39d0216e18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894ed48ecdad45b0b2974f39d0216e18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37557 } }
I20260926 15:20:08.633688 28527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 894ed48ecdad45b0b2974f39d0216e18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:20:08.633811 28527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 894ed48ecdad45b0b2974f39d0216e18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894ed48ecdad45b0b2974f39d0216e18, State: Initialized, Role: FOLLOWER
I20260926 15:20:08.634361 28527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 894ed48ecdad45b0b2974f39d0216e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894ed48ecdad45b0b2974f39d0216e18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37557 } }
I20260926 15:20:08.634668 28527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 894ed48ecdad45b0b2974f39d0216e18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:20:08.634814 28527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 894ed48ecdad45b0b2974f39d0216e18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:20:08.635011 28527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 894ed48ecdad45b0b2974f39d0216e18 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:20:08.635324 28530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:08.629553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.249:42073 (local address 127.0.0.1:37557)
0926 15:20:08.629700 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:08.629704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:08.629961 (+ 257us) server_negotiation.cc:407] Connection header received
0926 15:20:08.630071 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:08.630079 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:08.630146 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:08.630247 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:08.630592 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:08.631542 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:08.632844 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:08.633477 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:08.633594 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:08.634513 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:08.634520 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:08.634524 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:20:08.634567 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:08.634607 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:20:08.634626 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:08.634760 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:08.634954 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 15:20:08.635013 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:20:08.637215 28527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 894ed48ecdad45b0b2974f39d0216e18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894ed48ecdad45b0b2974f39d0216e18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37557 } }
I20260926 15:20:08.637599 28527 leader_election.cc:302] T 00000000000000000000000000000000 P 894ed48ecdad45b0b2974f39d0216e18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 894ed48ecdad45b0b2974f39d0216e18; no voters:
I20260926 15:20:08.638053 28527 leader_election.cc:288] T 00000000000000000000000000000000 P 894ed48ecdad45b0b2974f39d0216e18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:20:08.638168 28537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 894ed48ecdad45b0b2974f39d0216e18 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:20:08.638933 28537 raft_consensus.cc:686] T 00000000000000000000000000000000 P 894ed48ecdad45b0b2974f39d0216e18 [term 1 LEADER]: Becoming Leader. State: Replica: 894ed48ecdad45b0b2974f39d0216e18, State: Running, Role: LEADER
I20260926 15:20:08.639490 28537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 894ed48ecdad45b0b2974f39d0216e18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894ed48ecdad45b0b2974f39d0216e18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37557 } }
I20260926 15:20:08.643684 28527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 894ed48ecdad45b0b2974f39d0216e18 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:20:08.649617 28538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 894ed48ecdad45b0b2974f39d0216e18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "894ed48ecdad45b0b2974f39d0216e18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894ed48ecdad45b0b2974f39d0216e18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37557 } } }
I20260926 15:20:08.650122 28538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 894ed48ecdad45b0b2974f39d0216e18 [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:08.650314 28539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 894ed48ecdad45b0b2974f39d0216e18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 894ed48ecdad45b0b2974f39d0216e18. Latest consensus state: current_term: 1 leader_uuid: "894ed48ecdad45b0b2974f39d0216e18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894ed48ecdad45b0b2974f39d0216e18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37557 } } }
I20260926 15:20:08.650645 28539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 894ed48ecdad45b0b2974f39d0216e18 [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:08.653278 28550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:20:08.657729 28550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:20:08.664286 28555 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 15:20:08.666745 28550 catalog_manager.cc:1347] Generated new cluster ID: 18cfd924c85b443ba8ef99857d032f16
I20260926 15:20:08.666857 28550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:20:08.677886 28550 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:20:08.678861 28550 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:20:08.684161 28550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 894ed48ecdad45b0b2974f39d0216e18: Generated new TSK 0
I20260926 15:20:08.685030 28550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:20:08.701090 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 28460
W20260926 15:20:08.702057 28522 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.249:46759
W20260926 15:20:08.702628 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 28460: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:20:08.702755 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 28460
W20260926 15:20:08.718458 28445 connection.cc:535] client connection to 127.0.0.1:37557 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37557): Connection reset by peer (error 104)
2026-09-26T15:20:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1680 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:20:08 dist-test-slave-j6q7 krb5kdc[28559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:20:08 dist-test-slave-j6q7 krb5kdc[28559](info): set up 2 sockets
Sep 26 15:20:08 dist-test-slave-j6q7 krb5kdc[28559](info): commencing operation
krb5kdc: starting...
W20260926 15:20:09.700909 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:20:09 dist-test-slave-j6q7 krb5kdc[28559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:20:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:20:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:20:09.888952 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36217
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:20:10.174482 28572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:20:10.174777 28572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:20:10.174932 28572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:20:10.181546 28572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:20:10.181685 28572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:20:10.181804 28572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:20:10.181905 28572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:20:10.181981 28572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:20:10.190827 28572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36217
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:20:10.192022 28572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:20:10.193272 28572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:20:10.203744 28580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:10.203749 28577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:10.203791 28578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:20:10.204646 28572 server_base.cc:1034] running on GCE node
I20260926 15:20:10.205387 28572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:20:10.206653 28572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:20:10.207851 28572 hybrid_clock.cc:648] HybridClock initialized: now 1790436010207804 us; error 41 us; skew 500 ppm
I20260926 15:20:10.208159 28572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:20:10 dist-test-slave-j6q7 krb5kdc[28559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:20:10.217656 28572 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:20:10.218904 28572 webserver.cc:460] Webserver started at http://127.0.0.1:36239/ using document root <none> and password file <none>
I20260926 15:20:10.219466 28572 fs_manager.cc:359] Metadata directory not provided
I20260926 15:20:10.219597 28572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:20:10.219889 28572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:20:10.222622 28572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "083f1ad753824d92b69114aaa97c44a8"
format_stamp: "Formatted at 2026-09-26 15:20:10 on dist-test-slave-j6q7"
server_key: "de58468ed40944ed2a7f8ee075d23ca1"
server_key_iv: "134a895f122b963d24a66cf5344e607e"
server_key_version: "encryptionkey@0"
I20260926 15:20:10.223374 28572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "083f1ad753824d92b69114aaa97c44a8"
format_stamp: "Formatted at 2026-09-26 15:20:10 on dist-test-slave-j6q7"
server_key: "de58468ed40944ed2a7f8ee075d23ca1"
server_key_iv: "134a895f122b963d24a66cf5344e607e"
server_key_version: "encryptionkey@0"
I20260926 15:20:10.227968 28572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260926 15:20:10.230834 28587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:20:10.231904 28572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 15:20:10.232074 28572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "083f1ad753824d92b69114aaa97c44a8"
format_stamp: "Formatted at 2026-09-26 15:20:10 on dist-test-slave-j6q7"
server_key: "de58468ed40944ed2a7f8ee075d23ca1"
server_key_iv: "134a895f122b963d24a66cf5344e607e"
server_key_version: "encryptionkey@0"
I20260926 15:20:10.232251 28572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:20:10.237128 28572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:20:10.240226 28572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:20:10.240497 28572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:20:10.257989 28572 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36217
I20260926 15:20:10.258049 28638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36217 every 8 connection(s)
I20260926 15:20:10.259068 28572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:20:10.262207 28639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:20:10.265542 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 28572
I20260926 15:20:10.265720 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:20:10.266245 6983 external_mini_cluster.cc:1428] Setting key f4726ca4fe236ec70055a4ca5ff8168b
I20260926 15:20:10.272444 28642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:10.267687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.249:56523 (local address 127.0.0.1:36217)
0926 15:20:10.268275 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:10.268285 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:10.268318 (+ 33us) server_negotiation.cc:407] Connection header received
0926 15:20:10.269706 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:10.269723 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:10.270006 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:10.270326 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:10.271212 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:10.271217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:10.271230 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:20:10.271240 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:10.271283 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:20:10.271301 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:10.271489 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:10.271604 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 15:20:10.271744 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":253,"threads_started":1}
I20260926 15:20:10.272694 28639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 083f1ad753824d92b69114aaa97c44a8: Bootstrap starting.
I20260926 15:20:10.275758 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:20:10.276021 28639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 083f1ad753824d92b69114aaa97c44a8: Neither blocks nor log segments found. Creating new log.
I20260926 15:20:10.276926 28639 log.cc:824] T 00000000000000000000000000000000 P 083f1ad753824d92b69114aaa97c44a8: Log is configured to *not* fsync() on all Append() calls
I20260926 15:20:10.279395 28639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 083f1ad753824d92b69114aaa97c44a8: No bootstrap required, opened a new log
I20260926 15:20:10.281728 28642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:10.279598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.249:51587 (local address 127.0.0.1:36217)
0926 15:20:10.279727 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:10.279734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:10.280069 (+ 335us) server_negotiation.cc:407] Connection header received
0926 15:20:10.280223 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:10.280234 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:10.280280 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:10.280372 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:10.281200 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:10.281205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:10.281210 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:20:10.281225 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:10.281272 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:20:10.281283 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:10.281395 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:10.281472 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 15:20:10.281519 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 15:20:10.286445 28639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 083f1ad753824d92b69114aaa97c44a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083f1ad753824d92b69114aaa97c44a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36217 } }
I20260926 15:20:10.286742 28639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 083f1ad753824d92b69114aaa97c44a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:20:10.286827 28639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 083f1ad753824d92b69114aaa97c44a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083f1ad753824d92b69114aaa97c44a8, State: Initialized, Role: FOLLOWER
I20260926 15:20:10.287339 28639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 083f1ad753824d92b69114aaa97c44a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083f1ad753824d92b69114aaa97c44a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36217 } }
I20260926 15:20:10.287606 28639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 083f1ad753824d92b69114aaa97c44a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:20:10.287736 28639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 083f1ad753824d92b69114aaa97c44a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:20:10.287878 28639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 083f1ad753824d92b69114aaa97c44a8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:20:10.289585 28639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 083f1ad753824d92b69114aaa97c44a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083f1ad753824d92b69114aaa97c44a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36217 } }
I20260926 15:20:10.289932 28639 leader_election.cc:302] T 00000000000000000000000000000000 P 083f1ad753824d92b69114aaa97c44a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 083f1ad753824d92b69114aaa97c44a8; no voters:
I20260926 15:20:10.290373 28639 leader_election.cc:288] T 00000000000000000000000000000000 P 083f1ad753824d92b69114aaa97c44a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:20:10.290462 28649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 083f1ad753824d92b69114aaa97c44a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:20:10.291075 28649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 083f1ad753824d92b69114aaa97c44a8 [term 1 LEADER]: Becoming Leader. State: Replica: 083f1ad753824d92b69114aaa97c44a8, State: Running, Role: LEADER
I20260926 15:20:10.291576 28649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 083f1ad753824d92b69114aaa97c44a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083f1ad753824d92b69114aaa97c44a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36217 } }
I20260926 15:20:10.292119 28639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 083f1ad753824d92b69114aaa97c44a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:20:10.294154 28650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 083f1ad753824d92b69114aaa97c44a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "083f1ad753824d92b69114aaa97c44a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083f1ad753824d92b69114aaa97c44a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36217 } } }
I20260926 15:20:10.294181 28651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 083f1ad753824d92b69114aaa97c44a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 083f1ad753824d92b69114aaa97c44a8. Latest consensus state: current_term: 1 leader_uuid: "083f1ad753824d92b69114aaa97c44a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083f1ad753824d92b69114aaa97c44a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36217 } } }
I20260926 15:20:10.294591 28650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 083f1ad753824d92b69114aaa97c44a8 [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:10.294591 28651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 083f1ad753824d92b69114aaa97c44a8 [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:10.296979 28654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:20:10.300475 28654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:20:10.308955 28654 catalog_manager.cc:1347] Generated new cluster ID: eed9b60c04e6408eb89987dce2fb8efe
I20260926 15:20:10.309058 28654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:20:10.326913 28654 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:20:10.327235 28654 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:20:10.333081 28654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 083f1ad753824d92b69114aaa97c44a8: Generated new TSK 0
I20260926 15:20:10.333781 28654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:20:10.346836 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 28572
W20260926 15:20:10.347668 28634 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.249:56523
W20260926 15:20:10.348199 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 28572: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:20:10.348338 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 28572
2026-09-26T15:20:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1641 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:20:10 dist-test-slave-j6q7 krb5kdc[28670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:20:10 dist-test-slave-j6q7 krb5kdc[28670](info): set up 2 sockets
Sep 26 15:20:10 dist-test-slave-j6q7 krb5kdc[28670](info): commencing operation
krb5kdc: starting...
W20260926 15:20:11.357071 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:20:11 dist-test-slave-j6q7 krb5kdc[28670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:20:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:20:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:20:11.553881 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:35367
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:41323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:35367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:20:11.854050 28683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:20:11.854302 28683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:20:11.854384 28683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:20:11.860901 28683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:20:11.860999 28683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:20:11.861069 28683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:20:11.861128 28683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:20:11.861191 28683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:20:11.870630 28683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:41323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:35367
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:35367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:20:11.871804 28683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:20:11.872974 28683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:20:11.883435 28691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:11.883486 28689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:11.883518 28688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:20:11.884526 28683 server_base.cc:1034] running on GCE node
I20260926 15:20:11.885283 28683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:20:11.886498 28683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:20:11.887696 28683 hybrid_clock.cc:648] HybridClock initialized: now 1790436011887663 us; error 43 us; skew 500 ppm
I20260926 15:20:11.887975 28683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:20:11 dist-test-slave-j6q7 krb5kdc[28670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436011, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:20:11.890717 28683 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:20:11.891824 28683 webserver.cc:460] Webserver started at http://127.6.209.254:40925/ using document root <none> and password file <none>
I20260926 15:20:11.892305 28683 fs_manager.cc:359] Metadata directory not provided
I20260926 15:20:11.892395 28683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:20:11.892596 28683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:20:11.894989 28683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "a05354b40575479eb83dcb2d6d26813f"
format_stamp: "Formatted at 2026-09-26 15:20:11 on dist-test-slave-j6q7"
server_key: "e1899a781bcba3e221f371dd264e8879"
server_key_iv: "e7bc7334715c9495b5449f614061e149"
server_key_version: "encryptionkey@0"
I20260926 15:20:11.895654 28683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "a05354b40575479eb83dcb2d6d26813f"
format_stamp: "Formatted at 2026-09-26 15:20:11 on dist-test-slave-j6q7"
server_key: "e1899a781bcba3e221f371dd264e8879"
server_key_iv: "e7bc7334715c9495b5449f614061e149"
server_key_version: "encryptionkey@0"
I20260926 15:20:11.899892 28683 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260926 15:20:11.902925 28698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:20:11.904099 28683 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:20:11.904222 28683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "a05354b40575479eb83dcb2d6d26813f"
format_stamp: "Formatted at 2026-09-26 15:20:11 on dist-test-slave-j6q7"
server_key: "e1899a781bcba3e221f371dd264e8879"
server_key_iv: "e7bc7334715c9495b5449f614061e149"
server_key_version: "encryptionkey@0"
I20260926 15:20:11.904371 28683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:20:11.926321 28683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:20:11.929826 28683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:20:11.930018 28683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:20:11.946362 28683 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:35367
I20260926 15:20:11.946384 28749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:35367 every 8 connection(s)
I20260926 15:20:11.947391 28683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:20:11.950515 28750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:20:11.950902 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 28683
I20260926 15:20:11.951045 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:20:11.951455 6983 external_mini_cluster.cc:1428] Setting key cba3b05231e189c80bd95bf70c64a253
I20260926 15:20:11.962885 28750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a05354b40575479eb83dcb2d6d26813f: Bootstrap starting.
Sep 26 15:20:11 dist-test-slave-j6q7 krb5kdc[28670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:20:11.965461 28750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a05354b40575479eb83dcb2d6d26813f: Neither blocks nor log segments found. Creating new log.
I20260926 15:20:11.966723 28750 log.cc:824] T 00000000000000000000000000000000 P a05354b40575479eb83dcb2d6d26813f: Log is configured to *not* fsync() on all Append() calls
I20260926 15:20:11.969688 28750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a05354b40575479eb83dcb2d6d26813f: No bootstrap required, opened a new log
I20260926 15:20:11.973197 28753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:11.952970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59913 (local address 127.6.209.254:35367)
0926 15:20:11.953661 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:11.953674 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:11.953728 (+ 54us) server_negotiation.cc:407] Connection header received
0926 15:20:11.956287 (+ 2559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:11.956311 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:11.956733 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:11.957145 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:11.958191 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:11.959961 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:11.961221 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:11.962119 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:11.962289 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:11.965706 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:11.965717 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:11.965733 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:11.965790 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:11.969386 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:11.970064 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:11.970075 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:11.970084 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:11.970268 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:11.970648 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:11.970652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:11.970654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:11.971223 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:11.971511 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:11.971795 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 15:20:11.972085 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":275,"threads_started":1}
I20260926 15:20:11.976533 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:20:11.978062 28750 raft_consensus.cc:348] T 00000000000000000000000000000000 P a05354b40575479eb83dcb2d6d26813f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05354b40575479eb83dcb2d6d26813f" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35367 } }
I20260926 15:20:11.978346 28750 raft_consensus.cc:374] T 00000000000000000000000000000000 P a05354b40575479eb83dcb2d6d26813f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:20:11.978425 28750 raft_consensus.cc:729] T 00000000000000000000000000000000 P a05354b40575479eb83dcb2d6d26813f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a05354b40575479eb83dcb2d6d26813f, State: Initialized, Role: FOLLOWER
I20260926 15:20:11.978894 28750 consensus_queue.cc:260] T 00000000000000000000000000000000 P a05354b40575479eb83dcb2d6d26813f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05354b40575479eb83dcb2d6d26813f" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35367 } }
I20260926 15:20:11.979094 28750 raft_consensus.cc:388] T 00000000000000000000000000000000 P a05354b40575479eb83dcb2d6d26813f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:20:11.979218 28750 raft_consensus.cc:482] T 00000000000000000000000000000000 P a05354b40575479eb83dcb2d6d26813f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:20:11.979363 28750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a05354b40575479eb83dcb2d6d26813f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:20:11.982043 28750 raft_consensus.cc:504] T 00000000000000000000000000000000 P a05354b40575479eb83dcb2d6d26813f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05354b40575479eb83dcb2d6d26813f" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35367 } }
I20260926 15:20:11.982586 28750 leader_election.cc:302] T 00000000000000000000000000000000 P a05354b40575479eb83dcb2d6d26813f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a05354b40575479eb83dcb2d6d26813f; no voters:
I20260926 15:20:11.983136 28750 leader_election.cc:288] T 00000000000000000000000000000000 P a05354b40575479eb83dcb2d6d26813f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:20:11.983254 28760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a05354b40575479eb83dcb2d6d26813f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:20:11.984936 28750 sys_catalog.cc:564] T 00000000000000000000000000000000 P a05354b40575479eb83dcb2d6d26813f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:20:11.985514 28760 raft_consensus.cc:686] T 00000000000000000000000000000000 P a05354b40575479eb83dcb2d6d26813f [term 1 LEADER]: Becoming Leader. State: Replica: a05354b40575479eb83dcb2d6d26813f, State: Running, Role: LEADER
I20260926 15:20:11.986096 28760 consensus_queue.cc:237] T 00000000000000000000000000000000 P a05354b40575479eb83dcb2d6d26813f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05354b40575479eb83dcb2d6d26813f" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35367 } }
I20260926 15:20:11.990101 28761 sys_catalog.cc:455] T 00000000000000000000000000000000 P a05354b40575479eb83dcb2d6d26813f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a05354b40575479eb83dcb2d6d26813f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05354b40575479eb83dcb2d6d26813f" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35367 } } }
I20260926 15:20:11.990517 28761 sys_catalog.cc:458] T 00000000000000000000000000000000 P a05354b40575479eb83dcb2d6d26813f [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:11.991004 28762 sys_catalog.cc:455] T 00000000000000000000000000000000 P a05354b40575479eb83dcb2d6d26813f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a05354b40575479eb83dcb2d6d26813f. Latest consensus state: current_term: 1 leader_uuid: "a05354b40575479eb83dcb2d6d26813f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05354b40575479eb83dcb2d6d26813f" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 35367 } } }
I20260926 15:20:11.991314 28762 sys_catalog.cc:458] T 00000000000000000000000000000000 P a05354b40575479eb83dcb2d6d26813f [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:11.997478 28753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:11.980419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37781 (local address 127.6.209.254:35367)
0926 15:20:11.980607 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:11.980620 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:11.981121 (+ 501us) server_negotiation.cc:407] Connection header received
0926 15:20:11.981272 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:11.981275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:11.981321 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:11.981395 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:11.983059 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:11.984298 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:11.985954 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:11.986543 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:11.986662 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:11.988748 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:11.988754 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:11.988763 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:11.988800 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:11.992662 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:11.993815 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:11.993823 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:11.993829 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:11.994000 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:11.994647 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:11.994654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:11.994660 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:11.995174 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:11.996593 (+ 1419us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:11.996789 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 15:20:11.997029 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 15:20:12.007555 28777 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a05354b40575479eb83dcb2d6d26813f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:20:12.007680 28777 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:20:12.007802 28769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:20:12.009388 28769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:20:12.016495 28769 catalog_manager.cc:1347] Generated new cluster ID: 2e3c7a2160b946f5ad835e157c10171b
I20260926 15:20:12.016585 28769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:20:12.031973 28769 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:20:12.033147 28769 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:20:12.038712 28769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a05354b40575479eb83dcb2d6d26813f: Generated new TSK 0
I20260926 15:20:12.039320 28769 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:20:12.048837 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 28683
I20260926 15:20:12.050165 28745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:20:12.131429 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 28683
W20260926 15:20:12.148011 28668 connection.cc:535] client connection to 127.6.209.254:35367 recv error: Network error: failed to read from TLS socket (remote: 127.6.209.254:35367): Connection reset by peer (error 104)
2026-09-26T15:20:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1787 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:20:12 dist-test-slave-j6q7 krb5kdc[28783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:20:12 dist-test-slave-j6q7 krb5kdc[28783](info): set up 2 sockets
Sep 26 15:20:12 dist-test-slave-j6q7 krb5kdc[28783](info): commencing operation
krb5kdc: starting...
W20260926 15:20:13.132766 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:20:13 dist-test-slave-j6q7 krb5kdc[28783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:20:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:20:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:20:13.325778 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:42035
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:46113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:42035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:20:13.611111 28796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:20:13.611372 28796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:20:13.611546 28796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:20:13.618494 28796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:20:13.618589 28796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:20:13.618633 28796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:20:13.618718 28796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:20:13.618780 28796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:20:13.627876 28796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:46113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:42035
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:42035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:20:13.629114 28796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:20:13.630254 28796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:20:13.641436 28801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:13.641462 28802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:13.641614 28804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:20:13.642732 28796 server_base.cc:1034] running on GCE node
I20260926 15:20:13.643384 28796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:20:13.644615 28796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:20:13.645828 28796 hybrid_clock.cc:648] HybridClock initialized: now 1790436013645791 us; error 55 us; skew 500 ppm
I20260926 15:20:13.646147 28796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:20:13 dist-test-slave-j6q7 krb5kdc[28783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436013, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:20:13.649165 28796 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:20:13.650525 28796 webserver.cc:460] Webserver started at http://127.6.209.254:40247/ using document root <none> and password file <none>
I20260926 15:20:13.651122 28796 fs_manager.cc:359] Metadata directory not provided
I20260926 15:20:13.651247 28796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:20:13.651502 28796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:20:13.654259 28796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "b7dac54953d64bf18a92da99a0a99892"
format_stamp: "Formatted at 2026-09-26 15:20:13 on dist-test-slave-j6q7"
server_key: "815b0ff8ac1b83700638f66b068f3df8"
server_key_iv: "9d7efafd1577b988412a87c132edf52f"
server_key_version: "encryptionkey@0"
I20260926 15:20:13.654945 28796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "b7dac54953d64bf18a92da99a0a99892"
format_stamp: "Formatted at 2026-09-26 15:20:13 on dist-test-slave-j6q7"
server_key: "815b0ff8ac1b83700638f66b068f3df8"
server_key_iv: "9d7efafd1577b988412a87c132edf52f"
server_key_version: "encryptionkey@0"
I20260926 15:20:13.659651 28796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 15:20:13.662817 28811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:20:13.663949 28796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 15:20:13.664099 28796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "b7dac54953d64bf18a92da99a0a99892"
format_stamp: "Formatted at 2026-09-26 15:20:13 on dist-test-slave-j6q7"
server_key: "815b0ff8ac1b83700638f66b068f3df8"
server_key_iv: "9d7efafd1577b988412a87c132edf52f"
server_key_version: "encryptionkey@0"
I20260926 15:20:13.664286 28796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:20:13.681454 28796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:20:13.684558 28796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:20:13.684856 28796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:20:13.700487 28796 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:42035
I20260926 15:20:13.700507 28862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:42035 every 8 connection(s)
I20260926 15:20:13.701656 28796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:20:13.702224 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 28796
I20260926 15:20:13.702404 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:20:13.702921 6983 external_mini_cluster.cc:1428] Setting key ab7125d28631a95a2c12dc412ca517d2
I20260926 15:20:13.705482 28863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:20:13.714340 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:13.704457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33871 (local address 127.6.209.254:42035)
0926 15:20:13.705185 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:13.705195 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:13.705233 (+ 38us) server_negotiation.cc:407] Connection header received
0926 15:20:13.706802 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:13.706821 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:13.707159 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:13.707446 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:13.707983 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:13.709157 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:13.710696 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:13.711213 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:13.711314 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:13.712354 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:13.712361 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:13.712371 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:20:13.712873 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:13.712951 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:20:13.712972 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:13.713086 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:13.713350 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 15:20:13.713515 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":332,"threads_started":1}
I20260926 15:20:13.716480 28863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7dac54953d64bf18a92da99a0a99892: Bootstrap starting.
I20260926 15:20:13.717531 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:20:13.719089 28863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7dac54953d64bf18a92da99a0a99892: Neither blocks nor log segments found. Creating new log.
I20260926 15:20:13.720079 28863 log.cc:824] T 00000000000000000000000000000000 P b7dac54953d64bf18a92da99a0a99892: Log is configured to *not* fsync() on all Append() calls
I20260926 15:20:13.722673 28863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7dac54953d64bf18a92da99a0a99892: No bootstrap required, opened a new log
I20260926 15:20:13.727236 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:13.721331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44749 (local address 127.6.209.254:42035)
0926 15:20:13.721468 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:13.721472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:13.721815 (+ 343us) server_negotiation.cc:407] Connection header received
0926 15:20:13.721935 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:13.721940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:13.722003 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:13.722097 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:13.722456 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:13.723395 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:13.724500 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:13.725221 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:13.725353 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:13.726580 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:13.726585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:13.726590 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:20:13.726606 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:13.726643 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:20:13.726659 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:13.726763 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:13.726871 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 15:20:13.726917 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 15:20:13.730543 28863 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7dac54953d64bf18a92da99a0a99892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7dac54953d64bf18a92da99a0a99892" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 42035 } }
I20260926 15:20:13.730815 28863 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7dac54953d64bf18a92da99a0a99892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:20:13.730877 28863 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7dac54953d64bf18a92da99a0a99892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7dac54953d64bf18a92da99a0a99892, State: Initialized, Role: FOLLOWER
I20260926 15:20:13.731293 28863 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7dac54953d64bf18a92da99a0a99892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7dac54953d64bf18a92da99a0a99892" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 42035 } }
I20260926 15:20:13.731474 28863 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7dac54953d64bf18a92da99a0a99892 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:20:13.731559 28863 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7dac54953d64bf18a92da99a0a99892 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:20:13.731664 28863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7dac54953d64bf18a92da99a0a99892 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:20:13.733433 28863 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7dac54953d64bf18a92da99a0a99892 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7dac54953d64bf18a92da99a0a99892" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 42035 } }
I20260926 15:20:13.733843 28863 leader_election.cc:302] T 00000000000000000000000000000000 P b7dac54953d64bf18a92da99a0a99892 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7dac54953d64bf18a92da99a0a99892; no voters:
I20260926 15:20:13.734364 28863 leader_election.cc:288] T 00000000000000000000000000000000 P b7dac54953d64bf18a92da99a0a99892 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:20:13.734424 28873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7dac54953d64bf18a92da99a0a99892 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:20:13.735018 28873 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7dac54953d64bf18a92da99a0a99892 [term 1 LEADER]: Becoming Leader. State: Replica: b7dac54953d64bf18a92da99a0a99892, State: Running, Role: LEADER
I20260926 15:20:13.735523 28873 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7dac54953d64bf18a92da99a0a99892 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7dac54953d64bf18a92da99a0a99892" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 42035 } }
I20260926 15:20:13.736150 28863 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7dac54953d64bf18a92da99a0a99892 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:20:13.738122 28874 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7dac54953d64bf18a92da99a0a99892 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7dac54953d64bf18a92da99a0a99892" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7dac54953d64bf18a92da99a0a99892" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 42035 } } }
I20260926 15:20:13.738441 28874 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7dac54953d64bf18a92da99a0a99892 [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:13.738399 28875 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7dac54953d64bf18a92da99a0a99892 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7dac54953d64bf18a92da99a0a99892. Latest consensus state: current_term: 1 leader_uuid: "b7dac54953d64bf18a92da99a0a99892" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7dac54953d64bf18a92da99a0a99892" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 42035 } } }
I20260926 15:20:13.738754 28875 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7dac54953d64bf18a92da99a0a99892 [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:13.739754 28881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:20:13.744712 28881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:20:13.752360 28881 catalog_manager.cc:1347] Generated new cluster ID: 10da6f992fec433d9504fea92b7da0db
I20260926 15:20:13.752451 28881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:20:13.771744 28881 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:20:13.772718 28881 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:20:13.781143 28881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7dac54953d64bf18a92da99a0a99892: Generated new TSK 0
I20260926 15:20:13.781762 28881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:20:13.790354 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 28796
W20260926 15:20:13.791313 28858 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33871
W20260926 15:20:13.791973 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 28796: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:20:13.792088 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 28796
W20260926 15:20:13.807065 28781 connection.cc:535] client connection to 127.6.209.254:42035 recv error: Network error: failed to read from TLS socket (remote: 127.6.209.254:42035): Connection reset by peer (error 104)
2026-09-26T15:20:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1658 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:20:13 dist-test-slave-j6q7 krb5kdc[28894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:20:13 dist-test-slave-j6q7 krb5kdc[28894](info): set up 2 sockets
Sep 26 15:20:13 dist-test-slave-j6q7 krb5kdc[28894](info): commencing operation
krb5kdc: starting...
W20260926 15:20:14.793092 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:20:14 dist-test-slave-j6q7 krb5kdc[28894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:20:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:20:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:20:14.991357 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:40529
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:40529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:20:15.281651 28907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:20:15.281898 28907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:20:15.282061 28907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:20:15.288339 28907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:20:15.288430 28907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:20:15.288532 28907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:20:15.288618 28907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:20:15.288707 28907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:20:15.297484 28907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:40529
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:40529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:20:15.298627 28907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:20:15.299793 28907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:20:15.310046 28912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:15.310118 28915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:15.310146 28913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:20:15.311038 28907 server_base.cc:1034] running on GCE node
I20260926 15:20:15.311738 28907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:20:15.312932 28907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:20:15.314137 28907 hybrid_clock.cc:648] HybridClock initialized: now 1790436015314104 us; error 40 us; skew 500 ppm
I20260926 15:20:15.314427 28907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:20:15 dist-test-slave-j6q7 krb5kdc[28894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436015, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:20:15.320909 28907 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:20:15.322228 28907 webserver.cc:460] Webserver started at http://127.6.209.254:41051/ using document root <none> and password file <none>
I20260926 15:20:15.322796 28907 fs_manager.cc:359] Metadata directory not provided
I20260926 15:20:15.322893 28907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:20:15.323117 28907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:20:15.325747 28907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "fff9cc54bd564c4b861ac5e892e05a08"
format_stamp: "Formatted at 2026-09-26 15:20:15 on dist-test-slave-j6q7"
server_key: "d3f03b481bde3c6c5275a261e3beeb9c"
server_key_iv: "076e7442f50561fd2bbb27221592093a"
server_key_version: "encryptionkey@0"
I20260926 15:20:15.326433 28907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "fff9cc54bd564c4b861ac5e892e05a08"
format_stamp: "Formatted at 2026-09-26 15:20:15 on dist-test-slave-j6q7"
server_key: "d3f03b481bde3c6c5275a261e3beeb9c"
server_key_iv: "076e7442f50561fd2bbb27221592093a"
server_key_version: "encryptionkey@0"
I20260926 15:20:15.330770 28907 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 15:20:15.333489 28922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:20:15.334561 28907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:20:15.334678 28907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "fff9cc54bd564c4b861ac5e892e05a08"
format_stamp: "Formatted at 2026-09-26 15:20:15 on dist-test-slave-j6q7"
server_key: "d3f03b481bde3c6c5275a261e3beeb9c"
server_key_iv: "076e7442f50561fd2bbb27221592093a"
server_key_version: "encryptionkey@0"
I20260926 15:20:15.334810 28907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:20:15.339771 28907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:20:15.343276 28907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:20:15.343480 28907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:20:15.358973 28907 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:40529
I20260926 15:20:15.359007 28973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:40529 every 8 connection(s)
I20260926 15:20:15.360150 28907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:20:15.361112 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 28907
I20260926 15:20:15.361296 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:20:15.361639 6983 external_mini_cluster.cc:1428] Setting key f9da116231f41646785f884bc994c1b6
I20260926 15:20:15.363651 28974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:20:15.367930 28977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:15.363005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50427 (local address 127.6.209.254:40529)
0926 15:20:15.363650 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:15.363659 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:15.363692 (+ 33us) server_negotiation.cc:407] Connection header received
0926 15:20:15.365082 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:15.365100 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:15.365421 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:15.365712 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:15.366639 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:15.366645 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:15.366655 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:20:15.366667 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:15.366727 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:20:15.366743 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:15.366900 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:15.367000 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 15:20:15.367166 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":301,"threads_started":1}
I20260926 15:20:15.370910 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:20:15.373175 28974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fff9cc54bd564c4b861ac5e892e05a08: Bootstrap starting.
I20260926 15:20:15.375995 28974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fff9cc54bd564c4b861ac5e892e05a08: Neither blocks nor log segments found. Creating new log.
I20260926 15:20:15.376720 28977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:15.374477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34305 (local address 127.6.209.254:40529)
0926 15:20:15.374589 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:15.374600 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:15.375002 (+ 402us) server_negotiation.cc:407] Connection header received
0926 15:20:15.375143 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:15.375147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:15.375187 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:15.375292 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:15.376140 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:15.376145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:15.376153 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:20:15.376170 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:15.376209 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:20:15.376223 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:15.376343 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:15.376455 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 15:20:15.376515 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 15:20:15.377125 28974 log.cc:824] T 00000000000000000000000000000000 P fff9cc54bd564c4b861ac5e892e05a08: Log is configured to *not* fsync() on all Append() calls
I20260926 15:20:15.379254 28974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fff9cc54bd564c4b861ac5e892e05a08: No bootstrap required, opened a new log
I20260926 15:20:15.385727 28974 raft_consensus.cc:348] T 00000000000000000000000000000000 P fff9cc54bd564c4b861ac5e892e05a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff9cc54bd564c4b861ac5e892e05a08" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40529 } }
I20260926 15:20:15.386010 28974 raft_consensus.cc:374] T 00000000000000000000000000000000 P fff9cc54bd564c4b861ac5e892e05a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:20:15.386082 28974 raft_consensus.cc:729] T 00000000000000000000000000000000 P fff9cc54bd564c4b861ac5e892e05a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fff9cc54bd564c4b861ac5e892e05a08, State: Initialized, Role: FOLLOWER
I20260926 15:20:15.386514 28974 consensus_queue.cc:260] T 00000000000000000000000000000000 P fff9cc54bd564c4b861ac5e892e05a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff9cc54bd564c4b861ac5e892e05a08" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40529 } }
I20260926 15:20:15.386685 28974 raft_consensus.cc:388] T 00000000000000000000000000000000 P fff9cc54bd564c4b861ac5e892e05a08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:20:15.386767 28974 raft_consensus.cc:482] T 00000000000000000000000000000000 P fff9cc54bd564c4b861ac5e892e05a08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:20:15.386873 28974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fff9cc54bd564c4b861ac5e892e05a08 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:20:15.388752 28974 raft_consensus.cc:504] T 00000000000000000000000000000000 P fff9cc54bd564c4b861ac5e892e05a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff9cc54bd564c4b861ac5e892e05a08" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40529 } }
I20260926 15:20:15.389137 28974 leader_election.cc:302] T 00000000000000000000000000000000 P fff9cc54bd564c4b861ac5e892e05a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fff9cc54bd564c4b861ac5e892e05a08; no voters:
I20260926 15:20:15.389539 28974 leader_election.cc:288] T 00000000000000000000000000000000 P fff9cc54bd564c4b861ac5e892e05a08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:20:15.389652 28984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fff9cc54bd564c4b861ac5e892e05a08 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:20:15.390220 28984 raft_consensus.cc:686] T 00000000000000000000000000000000 P fff9cc54bd564c4b861ac5e892e05a08 [term 1 LEADER]: Becoming Leader. State: Replica: fff9cc54bd564c4b861ac5e892e05a08, State: Running, Role: LEADER
I20260926 15:20:15.390640 28984 consensus_queue.cc:237] T 00000000000000000000000000000000 P fff9cc54bd564c4b861ac5e892e05a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff9cc54bd564c4b861ac5e892e05a08" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40529 } }
I20260926 15:20:15.391175 28974 sys_catalog.cc:564] T 00000000000000000000000000000000 P fff9cc54bd564c4b861ac5e892e05a08 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:20:15.393064 28986 sys_catalog.cc:455] T 00000000000000000000000000000000 P fff9cc54bd564c4b861ac5e892e05a08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fff9cc54bd564c4b861ac5e892e05a08. Latest consensus state: current_term: 1 leader_uuid: "fff9cc54bd564c4b861ac5e892e05a08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff9cc54bd564c4b861ac5e892e05a08" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40529 } } }
I20260926 15:20:15.393460 28986 sys_catalog.cc:458] T 00000000000000000000000000000000 P fff9cc54bd564c4b861ac5e892e05a08 [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:15.394811 28990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:20:15.394718 28985 sys_catalog.cc:455] T 00000000000000000000000000000000 P fff9cc54bd564c4b861ac5e892e05a08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fff9cc54bd564c4b861ac5e892e05a08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff9cc54bd564c4b861ac5e892e05a08" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40529 } } }
I20260926 15:20:15.394953 28985 sys_catalog.cc:458] T 00000000000000000000000000000000 P fff9cc54bd564c4b861ac5e892e05a08 [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:15.399966 28990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:20:15.407305 28990 catalog_manager.cc:1347] Generated new cluster ID: 1e96acdfa29e47b9852b594427324afc
I20260926 15:20:15.407405 28990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:20:15.442322 28990 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:20:15.442651 28990 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:20:15.456065 28990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fff9cc54bd564c4b861ac5e892e05a08: Generated new TSK 0
I20260926 15:20:15.456890 28990 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:20:15.474192 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 28907
W20260926 15:20:15.475016 28969 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50427
W20260926 15:20:15.475657 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 28907: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:20:15.475785 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 28907
2026-09-26T15:20:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1684 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:20:15 dist-test-slave-j6q7 krb5kdc[29005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:20:15 dist-test-slave-j6q7 krb5kdc[29005](info): set up 2 sockets
Sep 26 15:20:15 dist-test-slave-j6q7 krb5kdc[29005](info): commencing operation
krb5kdc: starting...
W20260926 15:20:16.464859 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:20:16 dist-test-slave-j6q7 krb5kdc[29005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:20:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:20:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:20:16.665706 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:46037
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:41889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:46037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:20:16.963714 29018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:20:16.963963 29018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:20:16.964104 29018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:20:16.970083 29018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:20:16.970197 29018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:20:16.970289 29018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:20:16.970376 29018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:20:16.970450 29018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:20:16.978557 29018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:41889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:46037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:46037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:20:16.979595 29018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:20:16.980695 29018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:20:16.991008 29023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:16.991076 29024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:16.991045 29026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:20:16.991989 29018 server_base.cc:1034] running on GCE node
I20260926 15:20:16.992628 29018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:20:16.994156 29018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:20:16.995370 29018 hybrid_clock.cc:648] HybridClock initialized: now 1790436016995314 us; error 42 us; skew 500 ppm
I20260926 15:20:16.995810 29018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:20:16 dist-test-slave-j6q7 krb5kdc[29005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436016, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:20:17.000113 29018 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:20:17.002182 29018 webserver.cc:460] Webserver started at http://127.6.209.254:45331/ using document root <none> and password file <none>
I20260926 15:20:17.002983 29018 fs_manager.cc:359] Metadata directory not provided
I20260926 15:20:17.003149 29018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:20:17.003582 29018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:20:17.007697 29018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "22ba6a35d4b6458ba8f0a807b360195d"
format_stamp: "Formatted at 2026-09-26 15:20:16 on dist-test-slave-j6q7"
server_key: "da06a7a61ba0df681f5d94cb91b15829"
server_key_iv: "98bfdd3d14e1c9bffdc1495452bca059"
server_key_version: "encryptionkey@0"
I20260926 15:20:17.008594 29018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "22ba6a35d4b6458ba8f0a807b360195d"
format_stamp: "Formatted at 2026-09-26 15:20:16 on dist-test-slave-j6q7"
server_key: "da06a7a61ba0df681f5d94cb91b15829"
server_key_iv: "98bfdd3d14e1c9bffdc1495452bca059"
server_key_version: "encryptionkey@0"
I20260926 15:20:17.014194 29018 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260926 15:20:17.018158 29033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:20:17.019381 29018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:20:17.019533 29018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "22ba6a35d4b6458ba8f0a807b360195d"
format_stamp: "Formatted at 2026-09-26 15:20:16 on dist-test-slave-j6q7"
server_key: "da06a7a61ba0df681f5d94cb91b15829"
server_key_iv: "98bfdd3d14e1c9bffdc1495452bca059"
server_key_version: "encryptionkey@0"
I20260926 15:20:17.019704 29018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:20:17.044061 29018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:20:17.047480 29018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:20:17.047719 29018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:20:17.062544 29018 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:46037
I20260926 15:20:17.062561 29084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:46037 every 8 connection(s)
I20260926 15:20:17.063572 29018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:20:17.066601 29085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:20:17.073256 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 29018
I20260926 15:20:17.073472 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:20:17.073875 6983 external_mini_cluster.cc:1428] Setting key f02c8d8c318af5423577bee1bb9b7203
I20260926 15:20:17.076133 29085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ba6a35d4b6458ba8f0a807b360195d: Bootstrap starting.
I20260926 15:20:17.079747 29085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22ba6a35d4b6458ba8f0a807b360195d: Neither blocks nor log segments found. Creating new log.
I20260926 15:20:17.080785 29085 log.cc:824] T 00000000000000000000000000000000 P 22ba6a35d4b6458ba8f0a807b360195d: Log is configured to *not* fsync() on all Append() calls
I20260926 15:20:17.083287 29085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ba6a35d4b6458ba8f0a807b360195d: No bootstrap required, opened a new log
Sep 26 15:20:17 dist-test-slave-j6q7 krb5kdc[29005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:20:17.091550 29085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22ba6a35d4b6458ba8f0a807b360195d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ba6a35d4b6458ba8f0a807b360195d" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 46037 } }
I20260926 15:20:17.091840 29085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22ba6a35d4b6458ba8f0a807b360195d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:20:17.091939 29085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22ba6a35d4b6458ba8f0a807b360195d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ba6a35d4b6458ba8f0a807b360195d, State: Initialized, Role: FOLLOWER
I20260926 15:20:17.092451 29085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22ba6a35d4b6458ba8f0a807b360195d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ba6a35d4b6458ba8f0a807b360195d" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 46037 } }
I20260926 15:20:17.092722 29085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22ba6a35d4b6458ba8f0a807b360195d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:20:17.092869 29085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22ba6a35d4b6458ba8f0a807b360195d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:20:17.093045 29085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22ba6a35d4b6458ba8f0a807b360195d [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:20:17.095302 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:17.075504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54725 (local address 127.6.209.254:46037)
0926 15:20:17.076246 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:17.076261 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:17.076308 (+ 47us) server_negotiation.cc:407] Connection header received
0926 15:20:17.078481 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:17.078504 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:17.078911 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:17.079299 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:17.080631 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:17.082299 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:17.083405 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:17.084378 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:17.084557 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:17.087984 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:17.087992 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:17.088006 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:17.088045 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:17.091420 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:17.091986 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:17.091994 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:17.092002 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:17.092175 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:17.092524 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:17.092529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:17.092532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:17.093184 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:17.093473 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:17.093725 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 15:20:17.094099 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":299,"threads_started":1}
I20260926 15:20:17.096048 29085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22ba6a35d4b6458ba8f0a807b360195d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ba6a35d4b6458ba8f0a807b360195d" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 46037 } }
I20260926 15:20:17.096560 29085 leader_election.cc:302] T 00000000000000000000000000000000 P 22ba6a35d4b6458ba8f0a807b360195d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22ba6a35d4b6458ba8f0a807b360195d; no voters:
I20260926 15:20:17.097259 29085 leader_election.cc:288] T 00000000000000000000000000000000 P 22ba6a35d4b6458ba8f0a807b360195d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:20:17.097373 29090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22ba6a35d4b6458ba8f0a807b360195d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:20:17.099009 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:20:17.099517 29085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22ba6a35d4b6458ba8f0a807b360195d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:20:17.101109 29090 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22ba6a35d4b6458ba8f0a807b360195d [term 1 LEADER]: Becoming Leader. State: Replica: 22ba6a35d4b6458ba8f0a807b360195d, State: Running, Role: LEADER
I20260926 15:20:17.101824 29090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22ba6a35d4b6458ba8f0a807b360195d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ba6a35d4b6458ba8f0a807b360195d" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 46037 } }
I20260926 15:20:17.104708 29091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22ba6a35d4b6458ba8f0a807b360195d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22ba6a35d4b6458ba8f0a807b360195d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ba6a35d4b6458ba8f0a807b360195d" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 46037 } } }
I20260926 15:20:17.105131 29091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22ba6a35d4b6458ba8f0a807b360195d [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:17.105778 29092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22ba6a35d4b6458ba8f0a807b360195d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22ba6a35d4b6458ba8f0a807b360195d. Latest consensus state: current_term: 1 leader_uuid: "22ba6a35d4b6458ba8f0a807b360195d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ba6a35d4b6458ba8f0a807b360195d" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 46037 } } }
I20260926 15:20:17.106199 29092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22ba6a35d4b6458ba8f0a807b360195d [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:17.122780 29096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:20:17.131198 29096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:20:17.144870 29096 catalog_manager.cc:1347] Generated new cluster ID: 4b183179015247708a58a381e9e3970d
I20260926 15:20:17.144963 29096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:20:17.150502 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:17.127764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55545 (local address 127.6.209.254:46037)
0926 15:20:17.127977 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:17.127984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:17.134377 (+ 6393us) server_negotiation.cc:407] Connection header received
0926 15:20:17.134523 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:17.134536 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:17.134616 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:17.134722 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:17.136751 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:17.137940 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:17.139585 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:17.140550 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:17.140672 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:17.143831 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:17.143837 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:17.143843 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:17.143878 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:17.147410 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:17.148054 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:17.148058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:17.148062 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:17.148193 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:17.148527 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:17.148532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:17.148535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:17.148872 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:17.149074 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:17.149989 (+ 915us) server_negotiation.cc:299] Negotiation successful
0926 15:20:17.150150 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 15:20:17.154601 29096 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:20:17.155920 29096 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:20:17.162133 29096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22ba6a35d4b6458ba8f0a807b360195d: Generated new TSK 0
I20260926 15:20:17.162696 29096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:20:17.169049 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 29018
I20260926 15:20:17.169992 29080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:20:17.252288 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 29018
2026-09-26T15:20:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1776 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:20:17 dist-test-slave-j6q7 krb5kdc[29118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:20:17 dist-test-slave-j6q7 krb5kdc[29118](info): set up 2 sockets
Sep 26 15:20:17 dist-test-slave-j6q7 krb5kdc[29118](info): commencing operation
krb5kdc: starting...
W20260926 15:20:18.249224 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:20:18 dist-test-slave-j6q7 krb5kdc[29118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:20:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:20:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:20:18.449968 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:40547
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:40547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:20:18.762583 29131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:20:18.762825 29131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:20:18.762909 29131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:20:18.769482 29131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:20:18.769569 29131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:20:18.769609 29131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:20:18.769665 29131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:20:18.769703 29131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:20:18.778146 29131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:40547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:40547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:20:18.779127 29131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:20:18.780210 29131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:20:18.790921 29137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:18.791016 29136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:18.790989 29139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:20:18.791698 29131 server_base.cc:1034] running on GCE node
I20260926 15:20:18.792423 29131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:20:18.793673 29131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:20:18.794888 29131 hybrid_clock.cc:648] HybridClock initialized: now 1790436018794863 us; error 56 us; skew 500 ppm
I20260926 15:20:18.795214 29131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:20:18 dist-test-slave-j6q7 krb5kdc[29118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436018, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:20:18.798374 29131 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:20:18.799641 29131 webserver.cc:460] Webserver started at http://127.6.209.254:44859/ using document root <none> and password file <none>
I20260926 15:20:18.800282 29131 fs_manager.cc:359] Metadata directory not provided
I20260926 15:20:18.800405 29131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:20:18.800678 29131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:20:18.803261 29131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "045b0a615f29431486de481057bdbeaf"
format_stamp: "Formatted at 2026-09-26 15:20:18 on dist-test-slave-j6q7"
server_key: "e12ff8cbb82ac3cf0f6734259c15b3e8"
server_key_iv: "507a4750cbd4ad65abf9b1e85d19a448"
server_key_version: "encryptionkey@0"
I20260926 15:20:18.803929 29131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "045b0a615f29431486de481057bdbeaf"
format_stamp: "Formatted at 2026-09-26 15:20:18 on dist-test-slave-j6q7"
server_key: "e12ff8cbb82ac3cf0f6734259c15b3e8"
server_key_iv: "507a4750cbd4ad65abf9b1e85d19a448"
server_key_version: "encryptionkey@0"
I20260926 15:20:18.808470 29131 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 15:20:18.811242 29146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:20:18.812242 29131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:20:18.812407 29131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "045b0a615f29431486de481057bdbeaf"
format_stamp: "Formatted at 2026-09-26 15:20:18 on dist-test-slave-j6q7"
server_key: "e12ff8cbb82ac3cf0f6734259c15b3e8"
server_key_iv: "507a4750cbd4ad65abf9b1e85d19a448"
server_key_version: "encryptionkey@0"
I20260926 15:20:18.812566 29131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:20:18.831956 29131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:20:18.835810 29131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:20:18.836071 29131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:20:18.852222 29131 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:40547
I20260926 15:20:18.852246 29197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:40547 every 8 connection(s)
I20260926 15:20:18.853282 29131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:20:18.853760 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 29131
I20260926 15:20:18.853935 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:20:18.854415 6983 external_mini_cluster.cc:1428] Setting key cb05d2e19200e9e5254d1e0fb63f99c2
I20260926 15:20:18.857209 29198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:20:18.867190 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:18.855961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33941 (local address 127.6.209.254:40547)
0926 15:20:18.856713 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:18.856726 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:18.856770 (+ 44us) server_negotiation.cc:407] Connection header received
0926 15:20:18.858837 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:18.858861 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:18.859241 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:18.859615 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:18.860055 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:18.861833 (+ 1778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:18.862869 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:18.863686 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:18.863836 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:18.864754 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:18.864764 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:18.864779 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:20:18.865394 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:18.865472 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:20:18.865496 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:18.865607 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:18.865784 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 15:20:18.866025 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":312,"threads_started":1}
I20260926 15:20:18.869010 29198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 045b0a615f29431486de481057bdbeaf: Bootstrap starting.
I20260926 15:20:18.870326 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:20:18.872154 29198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 045b0a615f29431486de481057bdbeaf: Neither blocks nor log segments found. Creating new log.
I20260926 15:20:18.873160 29198 log.cc:824] T 00000000000000000000000000000000 P 045b0a615f29431486de481057bdbeaf: Log is configured to *not* fsync() on all Append() calls
I20260926 15:20:18.875617 29198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 045b0a615f29431486de481057bdbeaf: No bootstrap required, opened a new log
I20260926 15:20:18.880223 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:18.874993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40533 (local address 127.6.209.254:40547)
0926 15:20:18.875216 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:18.875225 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:18.875728 (+ 503us) server_negotiation.cc:407] Connection header received
0926 15:20:18.875898 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:18.875906 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:18.875964 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:18.876069 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:18.876521 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:18.877247 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:18.878300 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:18.878650 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:18.878774 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:18.879612 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:18.879615 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:18.879618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:20:18.879633 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:18.879662 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:20:18.879674 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:18.879764 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:18.879883 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 15:20:18.879962 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 15:20:18.882759 29198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 045b0a615f29431486de481057bdbeaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045b0a615f29431486de481057bdbeaf" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40547 } }
I20260926 15:20:18.883173 29198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 045b0a615f29431486de481057bdbeaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:20:18.883307 29198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 045b0a615f29431486de481057bdbeaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 045b0a615f29431486de481057bdbeaf, State: Initialized, Role: FOLLOWER
I20260926 15:20:18.883788 29198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 045b0a615f29431486de481057bdbeaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045b0a615f29431486de481057bdbeaf" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40547 } }
I20260926 15:20:18.883977 29198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 045b0a615f29431486de481057bdbeaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:20:18.884057 29198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 045b0a615f29431486de481057bdbeaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:20:18.884171 29198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 045b0a615f29431486de481057bdbeaf [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:20:18.885885 29198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 045b0a615f29431486de481057bdbeaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045b0a615f29431486de481057bdbeaf" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40547 } }
I20260926 15:20:18.886270 29198 leader_election.cc:302] T 00000000000000000000000000000000 P 045b0a615f29431486de481057bdbeaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 045b0a615f29431486de481057bdbeaf; no voters:
I20260926 15:20:18.886816 29198 leader_election.cc:288] T 00000000000000000000000000000000 P 045b0a615f29431486de481057bdbeaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:20:18.886932 29208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 045b0a615f29431486de481057bdbeaf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:20:18.887578 29208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 045b0a615f29431486de481057bdbeaf [term 1 LEADER]: Becoming Leader. State: Replica: 045b0a615f29431486de481057bdbeaf, State: Running, Role: LEADER
I20260926 15:20:18.888015 29208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 045b0a615f29431486de481057bdbeaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045b0a615f29431486de481057bdbeaf" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40547 } }
I20260926 15:20:18.888515 29198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 045b0a615f29431486de481057bdbeaf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:20:18.890527 29210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 045b0a615f29431486de481057bdbeaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 045b0a615f29431486de481057bdbeaf. Latest consensus state: current_term: 1 leader_uuid: "045b0a615f29431486de481057bdbeaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045b0a615f29431486de481057bdbeaf" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40547 } } }
I20260926 15:20:18.890426 29209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 045b0a615f29431486de481057bdbeaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "045b0a615f29431486de481057bdbeaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045b0a615f29431486de481057bdbeaf" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 40547 } } }
I20260926 15:20:18.890988 29209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 045b0a615f29431486de481057bdbeaf [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:18.892414 29210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 045b0a615f29431486de481057bdbeaf [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:18.892498 29216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:20:18.899588 29216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:20:18.907477 29216 catalog_manager.cc:1347] Generated new cluster ID: 384e163cc82a431c864fd9f893a77666
I20260926 15:20:18.907580 29216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:20:18.915803 29216 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:20:18.917097 29216 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:20:18.922925 29216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 045b0a615f29431486de481057bdbeaf: Generated new TSK 0
I20260926 15:20:18.923512 29216 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:20:18.937278 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 29131
W20260926 15:20:18.938199 29193 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33941
W20260926 15:20:18.938773 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 29131: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:20:18.938886 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 29131
2026-09-26T15:20:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1685 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:20:19 dist-test-slave-j6q7 krb5kdc[29229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:20:19 dist-test-slave-j6q7 krb5kdc[29229](info): set up 2 sockets
Sep 26 15:20:19 dist-test-slave-j6q7 krb5kdc[29229](info): commencing operation
krb5kdc: starting...
W20260926 15:20:19.925339 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.006s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:20:20 dist-test-slave-j6q7 krb5kdc[29229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:20:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:20:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:20:20.145958 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:39949
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:43341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:39949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:20:20.432219 29242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:20:20.432485 29242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:20:20.432688 29242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:20:20.439383 29242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:20:20.439477 29242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:20:20.439524 29242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:20:20.439606 29242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:20:20.439682 29242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:20:20.448441 29242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:43341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:39949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:39949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:20:20.449520 29242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:20:20.450675 29242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:20:20.463665 29248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:20.463806 29250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:20.463976 29247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:20:20.465253 29242 server_base.cc:1034] running on GCE node
I20260926 15:20:20.465925 29242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:20:20.467157 29242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:20:20.468354 29242 hybrid_clock.cc:648] HybridClock initialized: now 1790436020468313 us; error 42 us; skew 500 ppm
I20260926 15:20:20.468631 29242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:20:20 dist-test-slave-j6q7 krb5kdc[29229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436020, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:20:20.477645 29242 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:20:20.479102 29242 webserver.cc:460] Webserver started at http://127.6.209.254:43831/ using document root <none> and password file <none>
I20260926 15:20:20.479704 29242 fs_manager.cc:359] Metadata directory not provided
I20260926 15:20:20.479799 29242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:20:20.480041 29242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:20:20.482668 29242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "645c433282f0402f9784afb1be282fad"
format_stamp: "Formatted at 2026-09-26 15:20:20 on dist-test-slave-j6q7"
server_key: "0cd56f12459d55496552783abb286d5e"
server_key_iv: "d4b247457bf794e541a4e669af4e6219"
server_key_version: "encryptionkey@0"
I20260926 15:20:20.483301 29242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "645c433282f0402f9784afb1be282fad"
format_stamp: "Formatted at 2026-09-26 15:20:20 on dist-test-slave-j6q7"
server_key: "0cd56f12459d55496552783abb286d5e"
server_key_iv: "d4b247457bf794e541a4e669af4e6219"
server_key_version: "encryptionkey@0"
I20260926 15:20:20.487850 29242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 15:20:20.490790 29257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:20:20.491890 29242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:20:20.492005 29242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "645c433282f0402f9784afb1be282fad"
format_stamp: "Formatted at 2026-09-26 15:20:20 on dist-test-slave-j6q7"
server_key: "0cd56f12459d55496552783abb286d5e"
server_key_iv: "d4b247457bf794e541a4e669af4e6219"
server_key_version: "encryptionkey@0"
I20260926 15:20:20.492139 29242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:20:20.497470 29242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:20:20.501413 29242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:20:20.501623 29242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:20:20.518214 29242 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:39949
I20260926 15:20:20.518281 29308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:39949 every 8 connection(s)
I20260926 15:20:20.519462 29242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:20:20.522733 29309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:20:20.524501 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 29242
I20260926 15:20:20.524659 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:20:20.525163 6983 external_mini_cluster.cc:1428] Setting key 26ff45386fb77f634f78521091024774
I20260926 15:20:20.533607 29312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:20.527003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48249 (local address 127.6.209.254:39949)
0926 15:20:20.527887 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:20.527900 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:20.528251 (+ 351us) server_negotiation.cc:407] Connection header received
0926 15:20:20.530333 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:20.530358 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:20.530786 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:20.531188 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:20.532034 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:20.532041 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:20.532056 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:20:20.532074 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:20.532132 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:20:20.532162 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:20.532341 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:20.532483 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 15:20:20.532645 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":420,"threads_started":1}
I20260926 15:20:20.534507 29309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 645c433282f0402f9784afb1be282fad: Bootstrap starting.
I20260926 15:20:20.536731 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:20:20.537547 29309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 645c433282f0402f9784afb1be282fad: Neither blocks nor log segments found. Creating new log.
I20260926 15:20:20.538615 29309 log.cc:824] T 00000000000000000000000000000000 P 645c433282f0402f9784afb1be282fad: Log is configured to *not* fsync() on all Append() calls
I20260926 15:20:20.541546 29309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 645c433282f0402f9784afb1be282fad: No bootstrap required, opened a new log
I20260926 15:20:20.543542 29312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:20.541417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38629 (local address 127.6.209.254:39949)
0926 15:20:20.541569 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:20.541574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:20.541936 (+ 362us) server_negotiation.cc:407] Connection header received
0926 15:20:20.542044 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:20.542050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:20.542095 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:20.542187 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:20.542952 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:20.542957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:20.542961 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:20:20.542980 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:20.543017 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:20:20.543033 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:20.543140 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:20.543212 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 15:20:20.543256 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 15:20:20.549180 29309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 645c433282f0402f9784afb1be282fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "645c433282f0402f9784afb1be282fad" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 39949 } }
I20260926 15:20:20.549443 29309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 645c433282f0402f9784afb1be282fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:20:20.549503 29309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 645c433282f0402f9784afb1be282fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 645c433282f0402f9784afb1be282fad, State: Initialized, Role: FOLLOWER
I20260926 15:20:20.549888 29309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 645c433282f0402f9784afb1be282fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "645c433282f0402f9784afb1be282fad" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 39949 } }
I20260926 15:20:20.550078 29309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 645c433282f0402f9784afb1be282fad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:20:20.550174 29309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 645c433282f0402f9784afb1be282fad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:20:20.550275 29309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 645c433282f0402f9784afb1be282fad [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:20:20.552338 29309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 645c433282f0402f9784afb1be282fad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "645c433282f0402f9784afb1be282fad" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 39949 } }
I20260926 15:20:20.552683 29309 leader_election.cc:302] T 00000000000000000000000000000000 P 645c433282f0402f9784afb1be282fad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 645c433282f0402f9784afb1be282fad; no voters:
I20260926 15:20:20.553130 29309 leader_election.cc:288] T 00000000000000000000000000000000 P 645c433282f0402f9784afb1be282fad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:20:20.553256 29319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 645c433282f0402f9784afb1be282fad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:20:20.553792 29319 raft_consensus.cc:686] T 00000000000000000000000000000000 P 645c433282f0402f9784afb1be282fad [term 1 LEADER]: Becoming Leader. State: Replica: 645c433282f0402f9784afb1be282fad, State: Running, Role: LEADER
I20260926 15:20:20.554261 29319 consensus_queue.cc:237] T 00000000000000000000000000000000 P 645c433282f0402f9784afb1be282fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "645c433282f0402f9784afb1be282fad" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 39949 } }
I20260926 15:20:20.554734 29309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 645c433282f0402f9784afb1be282fad [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:20:20.556949 29320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 645c433282f0402f9784afb1be282fad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "645c433282f0402f9784afb1be282fad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "645c433282f0402f9784afb1be282fad" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 39949 } } }
I20260926 15:20:20.557369 29320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 645c433282f0402f9784afb1be282fad [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:20.556859 29321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 645c433282f0402f9784afb1be282fad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 645c433282f0402f9784afb1be282fad. Latest consensus state: current_term: 1 leader_uuid: "645c433282f0402f9784afb1be282fad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "645c433282f0402f9784afb1be282fad" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 39949 } } }
I20260926 15:20:20.557806 29321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 645c433282f0402f9784afb1be282fad [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:20.559578 29327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:20:20.564602 29327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:20:20.571432 29327 catalog_manager.cc:1347] Generated new cluster ID: f2a56c3f56214756a2801c25bed6cc29
I20260926 15:20:20.571518 29327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:20:20.591858 29327 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:20:20.592196 29327 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:20:20.606592 29327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 645c433282f0402f9784afb1be282fad: Generated new TSK 0
I20260926 15:20:20.607220 29327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:20:20.618477 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 29242
W20260926 15:20:20.619392 29304 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48249
W20260926 15:20:20.619967 6983 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 29242: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:20:20.620127 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 29242
2026-09-26T15:20:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1681 ms)
[----------] 18 tests from AuthTokenIssuingTest (31119 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:20:20 dist-test-slave-j6q7 krb5kdc[29340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:20:20 dist-test-slave-j6q7 krb5kdc[29340](info): set up 2 sockets
Sep 26 15:20:20 dist-test-slave-j6q7 krb5kdc[29340](info): commencing operation
krb5kdc: starting...
W20260926 15:20:21.620983 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:20:21 dist-test-slave-j6q7 krb5kdc[29340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:20:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:20:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:20:21.841943 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:38959
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:38959,127.6.209.253:42265,127.6.209.252:42241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:20:22.132970 29353 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 15:20:22.133267 29353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:20:22.133332 29353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:20:22.133404 29353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:20:22.139815 29353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:20:22.139948 29353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:20:22.140089 29353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:20:22.140185 29353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:20:22.140280 29353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:20:22.149158 29353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:38959,127.6.209.253:42265,127.6.209.252:42241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:38959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:20:22.150341 29353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:20:22.151539 29353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:20:22.162429 29361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:22.162497 29359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:22.162497 29358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:20:22.163554 29353 server_base.cc:1034] running on GCE node
I20260926 15:20:22.164183 29353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:20:22.165400 29353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:20:22.166603 29353 hybrid_clock.cc:648] HybridClock initialized: now 1790436022166553 us; error 24 us; skew 500 ppm
I20260926 15:20:22.166900 29353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:20:22 dist-test-slave-j6q7 krb5kdc[29340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436022, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:20:22.169730 29353 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:20:22.170926 29353 webserver.cc:460] Webserver started at http://127.6.209.254:37893/ using document root <none> and password file <none>
I20260926 15:20:22.171535 29353 fs_manager.cc:359] Metadata directory not provided
I20260926 15:20:22.171650 29353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:20:22.171932 29353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:20:22.174536 29353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "453738e3a26047c5a01f41f8531cb803"
format_stamp: "Formatted at 2026-09-26 15:20:22 on dist-test-slave-j6q7"
server_key: "d9ed42c04f9377ed1af6db3f244608a5"
server_key_iv: "b905f85af9a7b646d30110b89a6196bd"
server_key_version: "encryptionkey@0"
I20260926 15:20:22.175212 29353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "453738e3a26047c5a01f41f8531cb803"
format_stamp: "Formatted at 2026-09-26 15:20:22 on dist-test-slave-j6q7"
server_key: "d9ed42c04f9377ed1af6db3f244608a5"
server_key_iv: "b905f85af9a7b646d30110b89a6196bd"
server_key_version: "encryptionkey@0"
I20260926 15:20:22.179714 29353 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 15:20:22.182780 29368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:20:22.183951 29353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:20:22.184124 29353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "453738e3a26047c5a01f41f8531cb803"
format_stamp: "Formatted at 2026-09-26 15:20:22 on dist-test-slave-j6q7"
server_key: "d9ed42c04f9377ed1af6db3f244608a5"
server_key_iv: "b905f85af9a7b646d30110b89a6196bd"
server_key_version: "encryptionkey@0"
I20260926 15:20:22.184331 29353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:20:22.200543 29353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:20:22.205138 29353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:20:22.205415 29353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:20:22.223133 29353 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:38959
I20260926 15:20:22.223161 29419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:38959 every 8 connection(s)
I20260926 15:20:22.224371 29353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:20:22.227718 29420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:20:22.231333 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 29353
I20260926 15:20:22.231529 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:20:22.232182 6983 external_mini_cluster.cc:1428] Setting key f3c768ea65b95dc730dcf1150e6c228f
I20260926 15:20:22.234027 29420 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38959 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:20:22 dist-test-slave-j6q7 krb5kdc[29340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436022, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.209.253@KRBTEST.COM; defaulting to no policy
I20260926 15:20:22.254750 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.236968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:38959 (local address 127.0.0.1:55334)
0926 15:20:22.237751 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.237797 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 15:20:22.239786 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:20:22.240565 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:20:22.240571 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:20:22.241282 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:20:22.241610 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:22.241625 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:22.243184 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:22.243190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:22.244211 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:22.244222 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:22.245150 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:22.245156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:22.245338 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:22.245975 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:20:22.245997 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:20:22.247996 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:20:22.250961 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:22.250975 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:22.250988 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:22.251536 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:22.251942 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:22.251946 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:22.251951 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:22.252144 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:22.252851 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:20:22.252861 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:20:22.253118 (+ 257us) client_negotiation.cc:769] Sending connection context
0926 15:20:22.253383 (+ 265us) client_negotiation.cc:240] Negotiation successful
0926 15:20:22.253910 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":77,"thread_start_us":361,"threads_started":1}
I20260926 15:20:22.254786 29423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.238199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55334 (local address 127.6.209.254:38959)
0926 15:20:22.238872 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:22.238886 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:22.238934 (+ 48us) server_negotiation.cc:407] Connection header received
0926 15:20:22.240142 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:22.240148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:22.240358 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:22.240549 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:22.241806 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:22.243007 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:22.244343 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:22.244925 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:22.245122 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:22.248196 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:22.248202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:22.248212 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:22.248235 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:22.250827 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:22.251657 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:22.251662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:22.251664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:22.251806 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:22.252252 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:22.252255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:22.252257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:22.252652 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:22.252885 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:22.253471 (+ 586us) server_negotiation.cc:299] Negotiation successful
0926 15:20:22.253799 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":452,"thread_start_us":307,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.253@KRBTEST.COM" created.
I20260926 15:20:22.257755 29420 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:20:22.258771 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.258440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42265 (local address 127.0.0.1:46772)
0926 15:20:22.258584 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.258684 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 15:20:22.259085 29372 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42265: connect: Connection refused (error 111)
I20260926 15:20:22.260579 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.260157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42265 (local address 127.0.0.1:46788)
0926 15:20:22.260321 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.260415 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 15:20:22.260758 29420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.253:42265: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42265: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.253.keytab.
Entry for principal kudu/127.6.209.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.253.keytab.
I20260926 15:20:22.281670 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.253:42265
--webserver_interface=127.6.209.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.253.keytab
--principal=kudu/127.6.209.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:38959,127.6.209.253:42265,127.6.209.252:42241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:20:22.310120 29420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } attempt: 1
I20260926 15:20:22.311143 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.310755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42265 (local address 127.0.0.1:46798)
0926 15:20:22.310927 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.311034 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 15:20:22.312036 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.311777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42265 (local address 127.0.0.1:46808)
0926 15:20:22.311880 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.311935 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 15:20:22.312253 29420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.253:42265: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42265: connect: Connection refused (error 111)
I20260926 15:20:22.380497 29420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } attempt: 2
I20260926 15:20:22.381706 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.381244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42265 (local address 127.0.0.1:46822)
0926 15:20:22.381483 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.381565 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 15:20:22.382649 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.382341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42265 (local address 127.0.0.1:46836)
0926 15:20:22.382460 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.382517 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 15:20:22.382891 29420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.253:42265: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42265: connect: Connection refused (error 111)
I20260926 15:20:22.474125 29420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } attempt: 3
I20260926 15:20:22.475440 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.474992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42265 (local address 127.0.0.1:46850)
0926 15:20:22.475192 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.475305 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 15:20:22.476506 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.476139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42265 (local address 127.0.0.1:46860)
0926 15:20:22.476289 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.476365 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 15:20:22.476743 29420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.253:42265: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42265: connect: Connection refused (error 111)
W20260926 15:20:22.575549 29427 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 15:20:22.575798 29427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:20:22.575852 29427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:20:22.576016 29427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:20:22.582417 29427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:20:22.582485 29427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:20:22.582531 29427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:20:22.582592 29427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:20:22.582669 29427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:20:22.591199 29427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:38959,127.6.209.253:42265,127.6.209.252:42241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.253.keytab
--principal=kudu/127.6.209.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.253:42265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.209.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:20:22.592340 29427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:20:22.593640 29427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:20:22.604157 29432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:22.604118 29433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:22.604202 29435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:20:22.605083 29427 server_base.cc:1034] running on GCE node
I20260926 15:20:22.605768 29427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:20:22.606974 29427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:20:22.608167 29427 hybrid_clock.cc:648] HybridClock initialized: now 1790436022608119 us; error 24 us; skew 500 ppm
I20260926 15:20:22.608471 29427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:20:22 dist-test-slave-j6q7 krb5kdc[29340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436022, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:20:22.611264 29427 init.cc:377] Logged in from keytab as kudu/127.6.209.253@KRBTEST.COM (short username kudu)
I20260926 15:20:22.612469 29427 webserver.cc:460] Webserver started at http://127.6.209.253:37069/ using document root <none> and password file <none>
I20260926 15:20:22.613111 29427 fs_manager.cc:359] Metadata directory not provided
I20260926 15:20:22.613227 29427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:20:22.613514 29427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:20:22.615909 29427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-1/data/instance:
uuid: "798abf7b4b9d4797992877fa0c78fb78"
format_stamp: "Formatted at 2026-09-26 15:20:22 on dist-test-slave-j6q7"
server_key: "48a624697e23aa689f88e32082eed6df"
server_key_iv: "633b6b7cb1c57bd866d16166e88f2e49"
server_key_version: "encryptionkey@0"
I20260926 15:20:22.616654 29427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-1/wal/instance:
uuid: "798abf7b4b9d4797992877fa0c78fb78"
format_stamp: "Formatted at 2026-09-26 15:20:22 on dist-test-slave-j6q7"
server_key: "48a624697e23aa689f88e32082eed6df"
server_key_iv: "633b6b7cb1c57bd866d16166e88f2e49"
server_key_version: "encryptionkey@0"
I20260926 15:20:22.620040 29420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } attempt: 4
I20260926 15:20:22.621141 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.620624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42265 (local address 127.0.0.1:46866)
0926 15:20:22.620938 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.621016 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260926 15:20:22.621273 29427 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 15:20:22.621992 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.621697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42265 (local address 127.0.0.1:46878)
0926 15:20:22.621817 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.621892 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 15:20:22.622169 29420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.253:42265: Network error: Client connection negotiation failed: client connection to 127.6.209.253:42265: connect: Connection refused (error 111)
I20260926 15:20:22.624096 29442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:20:22.625234 29427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:20:22.625381 29427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-1/wal
uuid: "798abf7b4b9d4797992877fa0c78fb78"
format_stamp: "Formatted at 2026-09-26 15:20:22 on dist-test-slave-j6q7"
server_key: "48a624697e23aa689f88e32082eed6df"
server_key_iv: "633b6b7cb1c57bd866d16166e88f2e49"
server_key_version: "encryptionkey@0"
I20260926 15:20:22.625557 29427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:20:22.637424 29427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:20:22.641302 29427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:20:22.641618 29427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:20:22.659766 29427 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.253:42265
I20260926 15:20:22.659782 29493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.253:42265 every 8 connection(s)
I20260926 15:20:22.661063 29427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-1/data/info.pb
I20260926 15:20:22.664077 29494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:20:22.669780 29494 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38959 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:20:22.670300 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 29427
I20260926 15:20:22.670495 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-1/wal/instance
I20260926 15:20:22.670854 6983 external_mini_cluster.cc:1428] Setting key 628c0e4354098042b5a2c90aa8c4fcf5
Sep 26 15:20:22 dist-test-slave-j6q7 krb5kdc[29340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436022, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.253@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:20:22.690518 29423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.672101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55338 (local address 127.6.209.254:38959)
0926 15:20:22.672319 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:22.672324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:22.673291 (+ 967us) server_negotiation.cc:407] Connection header received
0926 15:20:22.675731 (+ 2440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:22.675735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:22.675794 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:22.675862 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:22.677739 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:22.678341 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:22.680325 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:22.681090 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:22.681242 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:22.684260 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:22.684268 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:22.684275 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:22.684312 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:22.687574 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:22.688451 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:22.688458 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:22.688464 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:22.688615 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:22.689052 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:22.689062 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:22.689065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:22.689330 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:22.689518 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:22.690049 (+ 531us) server_negotiation.cc:299] Negotiation successful
0926 15:20:22.690232 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 15:20:22.691313 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.672403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:38959 (local address 127.0.0.1:55338)
0926 15:20:22.673122 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.673172 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 15:20:22.675477 (+ 2305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:20:22.676110 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:20:22.676123 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:20:22.676856 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:20:22.677601 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:22.677622 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:22.678478 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:22.678486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:22.680186 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:22.680198 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:22.681288 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:22.681292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:22.681445 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:22.682001 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:20:22.682019 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:20:22.684067 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:20:22.687689 (+ 3622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:22.687695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:22.687704 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:22.688296 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:22.688718 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:22.688721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:22.688723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:22.688954 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:22.689490 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:20:22.689496 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:20:22.689748 (+ 252us) client_negotiation.cc:769] Sending connection context
0926 15:20:22.689965 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 15:20:22.690353 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":244,"threads_started":1}
I20260926 15:20:22.692847 29494 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.209.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.252@KRBTEST.COM" created.
Sep 26 15:20:22 dist-test-slave-j6q7 krb5kdc[29340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436022, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.253@KRBTEST.COM for kudu/127.6.209.253@KRBTEST.COM
I20260926 15:20:22.706251 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.693518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42265 (local address 127.0.0.1:46884)
0926 15:20:22.693655 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.693673 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:20:22.693757 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:20:22.694704 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:20:22.694708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:20:22.695098 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:20:22.695220 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:22.695227 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:22.696568 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:22.696579 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:22.697650 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:22.697665 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:22.698395 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:22.698406 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:22.698616 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:22.699101 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:20:22.699124 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:20:22.701384 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:20:22.704110 (+ 2726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:22.704116 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:22.704119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:22.704422 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:22.704719 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:22.704725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:22.704730 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:22.704918 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:22.705437 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:20:22.705443 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:20:22.705563 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 15:20:22.705683 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 15:20:22.705895 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 15:20:22.706375 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.693756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46884 (local address 127.6.209.253:42265)
0926 15:20:22.694363 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:22.694368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:22.694389 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:20:22.694423 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:22.694428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:22.694576 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:22.694716 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:22.695379 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:22.696420 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:22.697787 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:22.698247 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:22.698334 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:22.701574 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:22.701583 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:22.701593 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:22.701618 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:22.703992 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:22.704528 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:22.704534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:22.704537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:22.704631 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:22.705003 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:22.705009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:22.705011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:22.705267 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:22.705495 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:22.705805 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 15:20:22.705982 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":361,"threads_started":1}
I20260926 15:20:22.708582 29494 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:20:22.709493 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.709097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:42241 (local address 127.0.0.1:41592)
0926 15:20:22.709237 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.709348 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 15:20:22.709692 29446 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
I20260926 15:20:22.711076 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.710796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:42241 (local address 127.0.0.1:41598)
0926 15:20:22.710928 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.710996 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 15:20:22.711305 29494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.252:42241: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.252.keytab.
Entry for principal kudu/127.6.209.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.252.keytab.
I20260926 15:20:22.721678 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.252:42241
--webserver_interface=127.6.209.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.252.keytab
--principal=kudu/127.6.209.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:38959,127.6.209.253:42265,127.6.209.252:42241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:20:22.760579 29494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } attempt: 1
I20260926 15:20:22.761696 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.761196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:42241 (local address 127.0.0.1:41604)
0926 15:20:22.761405 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.761522 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 15:20:22.762669 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.762304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:42241 (local address 127.0.0.1:41606)
0926 15:20:22.762447 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.762548 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 15:20:22.762861 29494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.252:42241: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
I20260926 15:20:22.831084 29494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } attempt: 2
I20260926 15:20:22.832458 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.831699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:42241 (local address 127.0.0.1:41616)
0926 15:20:22.832078 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.832269 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 15:20:22.833472 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.833126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:42241 (local address 127.0.0.1:41632)
0926 15:20:22.833275 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.833347 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 15:20:22.833650 29494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.252:42241: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
I20260926 15:20:22.921381 29420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } attempt: 5
I20260926 15:20:22.924849 29494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } attempt: 3
I20260926 15:20:22.925829 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.925329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:42241 (local address 127.0.0.1:41648)
0926 15:20:22.925535 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.925644 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 15:20:22.927608 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.927102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:42241 (local address 127.0.0.1:41654)
0926 15:20:22.927389 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.927476 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
W20260926 15:20:22.927836 29494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.252:42241: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
Sep 26 15:20:22 dist-test-slave-j6q7 krb5kdc[29340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436022, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for kudu/127.6.209.253@KRBTEST.COM
I20260926 15:20:22.934352 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.922222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46886 (local address 127.6.209.253:42265)
0926 15:20:22.922421 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:22.922429 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:22.922450 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:20:22.922512 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:22.922519 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:22.922612 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:22.922712 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:22.923355 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:22.924254 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:22.925533 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:22.925972 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:22.926099 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:22.929720 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:22.929724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:22.929727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:22.929752 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:22.932065 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:22.932862 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:22.932865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:22.932868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:22.932968 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:22.933331 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:22.933334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:22.933335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:22.933553 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:22.933645 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:22.934062 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 15:20:22.934192 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 15:20:22.934659 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.922068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42265 (local address 127.0.0.1:46886)
0926 15:20:22.922271 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.922291 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:20:22.922401 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:20:22.922743 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:20:22.922746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:20:22.923064 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:20:22.923221 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:22.923229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:22.924385 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:22.924388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:22.925391 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:22.925405 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:22.926130 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:22.926137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:22.926369 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:22.927006 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:20:22.927031 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:20:22.929576 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:20:22.932191 (+ 2615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:22.932198 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:22.932204 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:22.932715 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:22.933070 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:22.933074 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:22.933078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:22.933239 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:22.933661 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:20:22.933667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:20:22.933833 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 15:20:22.933997 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 15:20:22.934297 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 15:20:22.935550 29420 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:20:22.936436 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.936085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:42241 (local address 127.0.0.1:41668)
0926 15:20:22.936240 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.936305 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 15:20:22.937443 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.937146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:42241 (local address 127.0.0.1:41684)
0926 15:20:22.937281 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.937354 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 15:20:22.937672 29420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.252:42241: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
I20260926 15:20:22.988883 29420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } attempt: 1
I20260926 15:20:22.989797 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.989433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:42241 (local address 127.0.0.1:41692)
0926 15:20:22.989586 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.989671 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 15:20:22.990614 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:22.990380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:42241 (local address 127.0.0.1:41696)
0926 15:20:22.990483 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:22.990536 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 15:20:22.990854 29420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.252:42241: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
W20260926 15:20:23.019330 29501 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 15:20:23.019567 29501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:20:23.019621 29501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:20:23.019729 29501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:20:23.026213 29501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:20:23.026285 29501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:20:23.026436 29501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:20:23.026510 29501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:20:23.026613 29501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:20:23.035712 29501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:38959,127.6.209.253:42265,127.6.209.252:42241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.252.keytab
--principal=kudu/127.6.209.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.252:42241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.209.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:20:23.036778 29501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:20:23.037968 29501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:20:23.048988 29507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:23.048950 29509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:23.049023 29506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:20:23.049714 29501 server_base.cc:1034] running on GCE node
I20260926 15:20:23.050357 29501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:20:23.051505 29501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:20:23.052722 29501 hybrid_clock.cc:648] HybridClock initialized: now 1790436023052671 us; error 27 us; skew 500 ppm
I20260926 15:20:23.053031 29501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:20:23 dist-test-slave-j6q7 krb5kdc[29340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436023, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:20:23.055589 29501 init.cc:377] Logged in from keytab as kudu/127.6.209.252@KRBTEST.COM (short username kudu)
I20260926 15:20:23.056890 29501 webserver.cc:460] Webserver started at http://127.6.209.252:37987/ using document root <none> and password file <none>
I20260926 15:20:23.057497 29501 fs_manager.cc:359] Metadata directory not provided
I20260926 15:20:23.057619 29501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:20:23.057910 29501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:20:23.059060 29420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } attempt: 2
I20260926 15:20:23.060105 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:23.059597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:42241 (local address 127.0.0.1:41702)
0926 15:20:23.059809 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:23.059903 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 15:20:23.061105 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:23.060729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:42241 (local address 127.0.0.1:41708)
0926 15:20:23.060896 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:23.060972 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 15:20:23.061353 29420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.252:42241: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
I20260926 15:20:23.061203 29501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-2/data/instance:
uuid: "44bedcd77f034aa6b8153ec56f2dbfd2"
format_stamp: "Formatted at 2026-09-26 15:20:23 on dist-test-slave-j6q7"
server_key: "9877df2679948d7650a7727019688c50"
server_key_iv: "b3337a2cc15efb09581c033f3edac272"
server_key_version: "encryptionkey@0"
I20260926 15:20:23.061982 29501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-2/wal/instance:
uuid: "44bedcd77f034aa6b8153ec56f2dbfd2"
format_stamp: "Formatted at 2026-09-26 15:20:23 on dist-test-slave-j6q7"
server_key: "9877df2679948d7650a7727019688c50"
server_key_iv: "b3337a2cc15efb09581c033f3edac272"
server_key_version: "encryptionkey@0"
I20260926 15:20:23.066450 29501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:20:23.069471 29516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:20:23.070642 29501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:20:23.070789 29501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-2/wal
uuid: "44bedcd77f034aa6b8153ec56f2dbfd2"
format_stamp: "Formatted at 2026-09-26 15:20:23 on dist-test-slave-j6q7"
server_key: "9877df2679948d7650a7727019688c50"
server_key_iv: "b3337a2cc15efb09581c033f3edac272"
server_key_version: "encryptionkey@0"
I20260926 15:20:23.071027 29494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } attempt: 4
I20260926 15:20:23.070991 29501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:20:23.071913 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:23.071544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:42241 (local address 127.0.0.1:41724)
0926 15:20:23.071708 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:23.071778 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 15:20:23.072692 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:23.072433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:42241 (local address 127.0.0.1:41736)
0926 15:20:23.072555 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:23.072613 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 15:20:23.072960 29494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.252:42241: Network error: Client connection negotiation failed: client connection to 127.6.209.252:42241: connect: Connection refused (error 111)
I20260926 15:20:23.100009 29501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:20:23.103318 29501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:20:23.103555 29501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:20:23.120218 29501 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.252:42241
I20260926 15:20:23.120249 29567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.252:42241 every 8 connection(s)
I20260926 15:20:23.121464 29501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-2/data/info.pb
I20260926 15:20:23.124468 29568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:20:23.129431 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 29501
I20260926 15:20:23.129580 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435694426955-6983-0/minicluster-data/master-2/wal/instance
I20260926 15:20:23.129964 6983 external_mini_cluster.cc:1428] Setting key b25df50c53bea75c7a8d585a3342a67a
I20260926 15:20:23.130901 29568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38959 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:20:23 dist-test-slave-j6q7 krb5kdc[29340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436023, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.252@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
Sep 26 15:20:23 dist-test-slave-j6q7 krb5kdc[29340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.252@KRBTEST.COM
I20260926 15:20:23.149540 29423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:23.133220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55340 (local address 127.6.209.254:38959)
0926 15:20:23.133399 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:23.133405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:23.134053 (+ 648us) server_negotiation.cc:407] Connection header received
0926 15:20:23.134437 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:23.134442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:23.134519 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:23.134625 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:23.136227 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.137327 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.139095 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.139588 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.139692 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:23.143376 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:23.143382 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:23.143386 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:23.143413 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:23.146437 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:23.147240 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:23.147248 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:23.147255 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:23.147423 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:23.147833 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:23.147837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:23.147839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:23.148149 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:23.148269 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:23.148842 (+ 573us) server_negotiation.cc:299] Negotiation successful
0926 15:20:23.149061 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 15:20:23.150118 29571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:23.131553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41744 (local address 127.6.209.252:42241)
0926 15:20:23.132302 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:23.132312 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:23.132345 (+ 33us) server_negotiation.cc:407] Connection header received
0926 15:20:23.134235 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:23.134253 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:23.134494 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:23.134657 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:23.136231 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.137789 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.139255 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.139923 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.140027 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:23.144133 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:23.144144 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:23.144159 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:23.144189 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:23.147192 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:23.147843 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:23.147849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:23.147852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:23.147969 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:23.148274 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:23.148279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:23.148284 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:23.148588 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:23.148870 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:23.149110 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 15:20:23.149294 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":370,"threads_started":1}
I20260926 15:20:23.150156 29572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:23.133220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:38959 (local address 127.0.0.1:55340)
0926 15:20:23.133913 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:23.133956 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 15:20:23.134240 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:20:23.134696 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:20:23.134703 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:20:23.135521 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:20:23.136004 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:23.136026 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.137482 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.137486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:23.138970 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:23.138983 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.139767 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.139771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:23.139913 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:23.140564 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:20:23.140610 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:20:23.143167 (+ 2557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:20:23.146556 (+ 3389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:23.146565 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:23.146571 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:23.147129 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:23.147519 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:23.147523 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:23.147526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:23.147715 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:23.148258 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:20:23.148265 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:20:23.148498 (+ 233us) client_negotiation.cc:769] Sending connection context
0926 15:20:23.148715 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 15:20:23.149131 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":339,"threads_started":1}
I20260926 15:20:23.151584 29568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:20:23 dist-test-slave-j6q7 krb5kdc[29340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436023, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.252@KRBTEST.COM for kudu/127.6.209.253@KRBTEST.COM
Sep 26 15:20:23 dist-test-slave-j6q7 krb5kdc[29340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:20:23.163580 29572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:23.152251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:42265 (local address 127.0.0.1:46898)
0926 15:20:23.152458 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:23.152469 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:20:23.152569 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:20:23.153102 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:20:23.153107 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:20:23.153481 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:20:23.153580 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:23.153587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.154376 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.154381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:23.155105 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:23.155115 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.155704 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.155709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:23.155872 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:23.156346 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:20:23.156361 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:20:23.158298 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:20:23.161667 (+ 3369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:23.161672 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:23.161675 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:23.161976 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:23.162292 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:23.162297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:23.162300 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:23.162418 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:23.162834 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:20:23.162838 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:20:23.162979 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 15:20:23.163117 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 15:20:23.163328 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260926 15:20:23.163650 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:23.152208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46898 (local address 127.6.209.253:42265)
0926 15:20:23.152364 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:23.152370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:23.152539 (+ 169us) server_negotiation.cc:407] Connection header received
0926 15:20:23.152655 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:23.152660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:23.152761 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:23.152916 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:23.153704 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.154261 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.155226 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.155584 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.155697 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:23.158697 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:23.158701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:23.158704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:23.158733 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:23.161552 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:23.162077 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:23.162082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:23.162085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:23.162198 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:23.162511 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:23.162515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:23.162518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:23.162726 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:23.162864 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:23.163188 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 15:20:23.163323 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 15:20:23.164340 29568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:20:23.165336 29423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:23.152803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.6.209.254:38959)
0926 15:20:23.152974 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:23.152978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:23.152992 (+ 14us) server_negotiation.cc:407] Connection header received
0926 15:20:23.153114 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:23.153120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:23.153191 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:23.153278 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:23.154200 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.154773 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.155766 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.156285 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.156430 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:23.160435 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:23.160442 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:23.160448 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:23.160488 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:23.162846 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:23.163439 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:23.163444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:23.163447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:23.163575 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:23.164146 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:23.164151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:23.164154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:23.164422 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:23.164662 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:23.164969 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 15:20:23.165155 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 15:20:23.167562 29420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } attempt: 3
Sep 26 15:20:23 dist-test-slave-j6q7 krb5kdc[29340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436023, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.252@KRBTEST.COM for kudu/127.6.209.252@KRBTEST.COM
Sep 26 15:20:23 dist-test-slave-j6q7 krb5kdc[29340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.253@KRBTEST.COM
Sep 26 15:20:23 dist-test-slave-j6q7 krb5kdc[29340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436022, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for kudu/127.6.209.252@KRBTEST.COM
I20260926 15:20:23.177632 29571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:23.164947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.6.209.252:42241)
0926 15:20:23.165054 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:23.165058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:23.165631 (+ 573us) server_negotiation.cc:407] Connection header received
0926 15:20:23.165753 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:23.165757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:23.165801 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:23.165864 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:23.166498 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.167201 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.168451 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.169034 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.169138 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:23.172126 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:23.172131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:23.172136 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:23.172165 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:23.174959 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:23.175594 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:23.175599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:23.175602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:23.175712 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:23.176103 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:23.176107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:23.176110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:23.176495 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:23.176637 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:23.177077 (+ 440us) server_negotiation.cc:299] Negotiation successful
0926 15:20:23.177311 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 15:20:23.177955 29572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:23.164872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:42241 (local address 127.0.0.1:41746)
0926 15:20:23.165564 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:23.165575 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:20:23.165672 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:20:23.165923 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:20:23.165926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:20:23.166268 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:20:23.166387 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:23.166394 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.167346 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.167351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:23.168325 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:23.168340 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.169196 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.169204 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:23.169488 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:23.169999 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:20:23.170017 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:20:23.171979 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:20:23.175082 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:23.175086 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:23.175090 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:23.175493 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:23.175837 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:23.175844 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:23.175849 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:23.175994 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:23.176622 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:20:23.176628 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:20:23.176772 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 15:20:23.176953 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 15:20:23.177509 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613}
I20260926 15:20:23.182494 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:23.166690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46904 (local address 127.6.209.253:42265)
0926 15:20:23.166830 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:23.166838 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:23.166854 (+ 16us) server_negotiation.cc:407] Connection header received
0926 15:20:23.167023 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:23.167028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:23.167091 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:23.167181 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:23.168384 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.169286 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.170588 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.171139 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.171238 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:23.176442 (+ 5204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:23.176446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:23.176449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:23.176476 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:23.179932 (+ 3456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:23.180558 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:23.180563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:23.180567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:23.180698 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:23.181120 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:23.181126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:23.181131 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:23.181414 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:23.181658 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:23.181895 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 15:20:23.182087 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 15:20:23.183341 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:23.167979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:42241 (local address 127.0.0.1:41756)
0926 15:20:23.168129 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:23.168144 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:20:23.168227 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:20:23.170009 (+ 1782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:20:23.170013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:20:23.170419 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:20:23.170588 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:23.170598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.171918 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.171922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:23.173205 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:23.173219 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.174220 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.174226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:23.174403 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:23.174908 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:20:23.174928 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:20:23.177245 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:20:23.180887 (+ 3642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:23.180893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:23.180899 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:23.181294 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:23.181712 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:23.181717 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:23.181720 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:23.181874 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:23.182428 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:20:23.182433 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:20:23.182649 (+ 216us) client_negotiation.cc:769] Sending connection context
0926 15:20:23.182773 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 15:20:23.183010 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 15:20:23.183391 29573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:23.168230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41756 (local address 127.6.209.252:42241)
0926 15:20:23.169773 (+ 1543us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:23.169778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:23.169796 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:20:23.169831 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:23.169835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:23.169885 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:23.169980 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:23.170729 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.171805 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.173352 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.174073 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.174211 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:23.177485 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:23.177490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:23.177495 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:23.177523 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:23.180688 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:23.181420 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:23.181425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:23.181427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:23.181580 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:23.182004 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:23.182010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:23.182015 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:23.182317 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:23.182454 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:23.182862 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 15:20:23.183019 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1439,"thread_start_us":1267,"threads_started":1}
I20260926 15:20:23.184247 29568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44bedcd77f034aa6b8153ec56f2dbfd2: Bootstrap starting.
I20260926 15:20:23.186913 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:20:23.187776 29568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44bedcd77f034aa6b8153ec56f2dbfd2: Neither blocks nor log segments found. Creating new log.
I20260926 15:20:23.188170 29420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803: Bootstrap starting.
I20260926 15:20:23.188961 29568 log.cc:824] T 00000000000000000000000000000000 P 44bedcd77f034aa6b8153ec56f2dbfd2: Log is configured to *not* fsync() on all Append() calls
I20260926 15:20:23.191399 29420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803: Neither blocks nor log segments found. Creating new log.
I20260926 15:20:23.191744 29568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44bedcd77f034aa6b8153ec56f2dbfd2: No bootstrap required, opened a new log
I20260926 15:20:23.189898 29395 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "453738e3a26047c5a01f41f8531cb803"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55348
I20260926 15:20:23.192667 29420 log.cc:824] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803: Log is configured to *not* fsync() on all Append() calls
I20260926 15:20:23.195225 29420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803: No bootstrap required, opened a new log
I20260926 15:20:23.201128 29568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44bedcd77f034aa6b8153ec56f2dbfd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "453738e3a26047c5a01f41f8531cb803" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38959 } } peers { permanent_uuid: "798abf7b4b9d4797992877fa0c78fb78" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } } peers { permanent_uuid: "44bedcd77f034aa6b8153ec56f2dbfd2" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } }
I20260926 15:20:23.201471 29568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44bedcd77f034aa6b8153ec56f2dbfd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44bedcd77f034aa6b8153ec56f2dbfd2, State: Initialized, Role: FOLLOWER
I20260926 15:20:23.201828 29420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "453738e3a26047c5a01f41f8531cb803" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38959 } } peers { permanent_uuid: "798abf7b4b9d4797992877fa0c78fb78" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } } peers { permanent_uuid: "44bedcd77f034aa6b8153ec56f2dbfd2" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } }
I20260926 15:20:23.201898 29568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44bedcd77f034aa6b8153ec56f2dbfd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "453738e3a26047c5a01f41f8531cb803" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38959 } } peers { permanent_uuid: "798abf7b4b9d4797992877fa0c78fb78" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } } peers { permanent_uuid: "44bedcd77f034aa6b8153ec56f2dbfd2" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } }
I20260926 15:20:23.202116 29420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 453738e3a26047c5a01f41f8531cb803, State: Initialized, Role: FOLLOWER
I20260926 15:20:23.202580 29420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "453738e3a26047c5a01f41f8531cb803" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38959 } } peers { permanent_uuid: "798abf7b4b9d4797992877fa0c78fb78" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } } peers { permanent_uuid: "44bedcd77f034aa6b8153ec56f2dbfd2" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } }
I20260926 15:20:23.202662 29576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44bedcd77f034aa6b8153ec56f2dbfd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "453738e3a26047c5a01f41f8531cb803" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38959 } } peers { permanent_uuid: "798abf7b4b9d4797992877fa0c78fb78" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } } peers { permanent_uuid: "44bedcd77f034aa6b8153ec56f2dbfd2" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } } }
I20260926 15:20:23.203025 29576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44bedcd77f034aa6b8153ec56f2dbfd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:20:23.203716 29568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44bedcd77f034aa6b8153ec56f2dbfd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:20:23.203500 29577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "453738e3a26047c5a01f41f8531cb803" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38959 } } peers { permanent_uuid: "798abf7b4b9d4797992877fa0c78fb78" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } } peers { permanent_uuid: "44bedcd77f034aa6b8153ec56f2dbfd2" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } } }
I20260926 15:20:23.203845 29577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:20:23.204445 29420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:20:23.204483 29395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 15:20:23.204667 29395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:20:23.207481 29395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "453738e3a26047c5a01f41f8531cb803" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38959 } } peers { permanent_uuid: "798abf7b4b9d4797992877fa0c78fb78" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } } peers { permanent_uuid: "44bedcd77f034aa6b8153ec56f2dbfd2" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } }
W20260926 15:20:23.207773 29374 tablet.cc:2369] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:20:23.208693 29395 leader_election.cc:288] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803 [CANDIDATE]: Term 1 election: Requested vote from peers 798abf7b4b9d4797992877fa0c78fb78 (127.6.209.253:42265), 44bedcd77f034aa6b8153ec56f2dbfd2 (127.6.209.252:42241)
I20260926 15:20:23.209627 29543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "453738e3a26047c5a01f41f8531cb803" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "44bedcd77f034aa6b8153ec56f2dbfd2"
I20260926 15:20:23.210100 29543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44bedcd77f034aa6b8153ec56f2dbfd2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:20:23.213901 29543 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44bedcd77f034aa6b8153ec56f2dbfd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453738e3a26047c5a01f41f8531cb803 in term 1.
I20260926 15:20:23.214527 29372 leader_election.cc:302] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44bedcd77f034aa6b8153ec56f2dbfd2, 453738e3a26047c5a01f41f8531cb803; no voters:
I20260926 15:20:23.215003 29577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:20:23.247232 29577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803 [term 1 LEADER]: Becoming Leader. State: Replica: 453738e3a26047c5a01f41f8531cb803, State: Running, Role: LEADER
I20260926 15:20:23.247843 29577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "453738e3a26047c5a01f41f8531cb803" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38959 } } peers { permanent_uuid: "798abf7b4b9d4797992877fa0c78fb78" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } } peers { permanent_uuid: "44bedcd77f034aa6b8153ec56f2dbfd2" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } }
I20260926 15:20:23.246927 29501 master_runner.cc:186] Error getting master registration for 127.6.209.254:38959: OK, instance_id { permanent_uuid: "453738e3a26047c5a01f41f8531cb803" instance_seqno: 1790436022218108 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 15:20:23.258978 29501 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 15:20:23.259047 29606 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 44bedcd77f034aa6b8153ec56f2dbfd2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:20:23.259133 29606 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:20:23.262404 29593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 453738e3a26047c5a01f41f8531cb803. Latest consensus state: current_term: 1 leader_uuid: "453738e3a26047c5a01f41f8531cb803" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "453738e3a26047c5a01f41f8531cb803" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38959 } } peers { permanent_uuid: "798abf7b4b9d4797992877fa0c78fb78" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } } peers { permanent_uuid: "44bedcd77f034aa6b8153ec56f2dbfd2" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } } }
I20260926 15:20:23.262806 29593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803 [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:23.265465 29353 master_runner.cc:186] Error getting master registration for 127.6.209.253:42265: OK, instance_id { permanent_uuid: "798abf7b4b9d4797992877fa0c78fb78" instance_seqno: 1790436022655189 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 15:20:23.279330 29353 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 15:20:23.291139 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:23.218883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46912 (local address 127.6.209.253:42265)
0926 15:20:23.219058 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:23.219065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:23.242979 (+ 23914us) server_negotiation.cc:407] Connection header received
0926 15:20:23.247527 (+ 4548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:23.247532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:23.247603 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:23.247667 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:23.254076 (+ 6409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.255328 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.265764 (+ 10436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.266653 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.266771 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:23.279757 (+ 12986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:23.279763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:23.279771 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:23.279826 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:23.284224 (+ 4398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:23.288728 (+ 4504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:23.288736 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:23.288742 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:23.288933 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:23.289422 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:23.289427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:23.289431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:23.289833 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:23.290070 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:23.290583 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 15:20:23.290798 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 15:20:23.294746 29573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:23.241879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41772 (local address 127.6.209.252:42241)
0926 15:20:23.242059 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:23.242066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:23.245639 (+ 3573us) server_negotiation.cc:407] Connection header received
0926 15:20:23.249540 (+ 3901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:23.249545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:23.249628 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:23.249705 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:23.262615 (+ 12910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.263643 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.270063 (+ 6420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.270856 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.270961 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:23.287871 (+ 16910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:23.287878 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:23.287888 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:23.287932 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:23.291489 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:23.292195 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:23.292200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:23.292203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:23.292361 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:23.292780 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:23.292787 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:23.292792 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:23.293127 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:23.293326 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:23.294204 (+ 878us) server_negotiation.cc:299] Negotiation successful
0926 15:20:23.294429 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 15:20:23.296164 29423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:23.213058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55358 (local address 127.6.209.254:38959)
0926 15:20:23.213246 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:23.213265 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:23.241346 (+ 28081us) server_negotiation.cc:407] Connection header received
0926 15:20:23.243549 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:23.243554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:23.243621 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:23.243681 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:23.257294 (+ 13613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.258212 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.268010 (+ 9798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.268915 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.269037 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:23.285398 (+ 16361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:23.285408 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:23.285416 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:23.285464 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:23.292517 (+ 7053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:23.293235 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:23.293240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:23.293242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:23.293451 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:23.293837 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:23.293842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:23.293845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:23.294604 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:23.295360 (+ 756us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:23.295547 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 15:20:23.295787 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 15:20:23.297655 29609 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:20:23.297784 29609 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:20:23.297916 29610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:20:23.299322 29610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:20:23.306502 29543 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 44bedcd77f034aa6b8153ec56f2dbfd2 [term 1 FOLLOWER]: Refusing update from remote peer 453738e3a26047c5a01f41f8531cb803: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:20:23.307463 29593 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44bedcd77f034aa6b8153ec56f2dbfd2" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:20:23.314934 29576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44bedcd77f034aa6b8153ec56f2dbfd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 453738e3a26047c5a01f41f8531cb803. Latest consensus state: current_term: 1 leader_uuid: "453738e3a26047c5a01f41f8531cb803" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "453738e3a26047c5a01f41f8531cb803" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38959 } } peers { permanent_uuid: "798abf7b4b9d4797992877fa0c78fb78" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } } peers { permanent_uuid: "44bedcd77f034aa6b8153ec56f2dbfd2" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } } }
I20260926 15:20:23.315171 29576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44bedcd77f034aa6b8153ec56f2dbfd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:20:23.316777 29612 mvcc.cc:205] Tried to move back new op lower bound from 7333625951453646848 to 7333625951229132800. Current Snapshot: MvccSnapshot[applied={T|T < 7333625951453646848}]
I20260926 15:20:23.318553 29613 mvcc.cc:205] Tried to move back new op lower bound from 7333625951453646848 to 7333625951229132800. Current Snapshot: MvccSnapshot[applied={T|T < 7333625951453646848}]
I20260926 15:20:23.322633 29577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "453738e3a26047c5a01f41f8531cb803" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "453738e3a26047c5a01f41f8531cb803" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38959 } } peers { permanent_uuid: "798abf7b4b9d4797992877fa0c78fb78" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } } peers { permanent_uuid: "44bedcd77f034aa6b8153ec56f2dbfd2" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } } }
I20260926 15:20:23.323007 29577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803 [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:23.323709 29576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44bedcd77f034aa6b8153ec56f2dbfd2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "453738e3a26047c5a01f41f8531cb803" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "453738e3a26047c5a01f41f8531cb803" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38959 } } peers { permanent_uuid: "798abf7b4b9d4797992877fa0c78fb78" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } } peers { permanent_uuid: "44bedcd77f034aa6b8153ec56f2dbfd2" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } } }
I20260926 15:20:23.323956 29576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44bedcd77f034aa6b8153ec56f2dbfd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:20:23.325919 29610 catalog_manager.cc:1347] Generated new cluster ID: 4cd8dc8e79f84936adf50a0858596341
I20260926 15:20:23.326013 29610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:20:23.347069 29610 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:20:23.348737 29610 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:20:23.359428 29610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803: Generated new TSK 0
I20260926 15:20:23.360917 29610 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:20:23.372167 29494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } attempt: 5
Sep 26 15:20:23 dist-test-slave-j6q7 krb5kdc[29340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436022, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.253@KRBTEST.COM for kudu/127.6.209.252@KRBTEST.COM
I20260926 15:20:23.392017 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:23.372646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:42241 (local address 127.0.0.1:41776)
0926 15:20:23.372891 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:23.372914 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 15:20:23.373058 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:20:23.373461 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:20:23.373467 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:20:23.373936 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:20:23.374124 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:23.374133 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.376291 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.376297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:23.377585 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:23.377597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.378897 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.378905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:23.379139 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:23.379650 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:20:23.379672 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:20:23.382525 (+ 2853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:20:23.386514 (+ 3989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:23.386522 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:23.386529 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:23.387398 (+ 869us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:23.387996 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:23.388002 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:23.388007 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:23.388271 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:23.389264 (+ 993us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:20:23.389289 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:20:23.389500 (+ 211us) client_negotiation.cc:769] Sending connection context
0926 15:20:23.389682 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 15:20:23.390600 (+ 918us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260926 15:20:23.392619 29573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:23.372683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41776 (local address 127.6.209.252:42241)
0926 15:20:23.372865 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:23.372870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:23.373012 (+ 142us) server_negotiation.cc:407] Connection header received
0926 15:20:23.373155 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:23.373160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:23.373235 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:23.373309 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:23.374252 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.375399 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.377748 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:23.378449 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:23.378579 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:23.382757 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:23.382765 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:23.382771 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:23.382806 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:23.386329 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:23.387567 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:23.387576 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:23.387581 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:23.387843 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:23.388399 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:23.388429 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:23.388435 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:23.389074 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:23.389911 (+ 837us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:23.390424 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 15:20:23.391190 (+ 766us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 15:20:23.398376 29494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 798abf7b4b9d4797992877fa0c78fb78: Bootstrap starting.
I20260926 15:20:23.402367 29494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 798abf7b4b9d4797992877fa0c78fb78: Neither blocks nor log segments found. Creating new log.
I20260926 15:20:23.404049 29494 log.cc:824] T 00000000000000000000000000000000 P 798abf7b4b9d4797992877fa0c78fb78: Log is configured to *not* fsync() on all Append() calls
I20260926 15:20:23.415493 29494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 798abf7b4b9d4797992877fa0c78fb78: No bootstrap required, opened a new log
I20260926 15:20:23.431923 29494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 798abf7b4b9d4797992877fa0c78fb78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "453738e3a26047c5a01f41f8531cb803" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38959 } } peers { permanent_uuid: "798abf7b4b9d4797992877fa0c78fb78" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } } peers { permanent_uuid: "44bedcd77f034aa6b8153ec56f2dbfd2" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } }
I20260926 15:20:23.432593 29494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 798abf7b4b9d4797992877fa0c78fb78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 798abf7b4b9d4797992877fa0c78fb78, State: Initialized, Role: FOLLOWER
I20260926 15:20:23.433418 29494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 798abf7b4b9d4797992877fa0c78fb78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "453738e3a26047c5a01f41f8531cb803" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38959 } } peers { permanent_uuid: "798abf7b4b9d4797992877fa0c78fb78" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } } peers { permanent_uuid: "44bedcd77f034aa6b8153ec56f2dbfd2" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } }
I20260926 15:20:23.444994 29469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "453738e3a26047c5a01f41f8531cb803" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "798abf7b4b9d4797992877fa0c78fb78"
I20260926 15:20:23.445338 29468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 798abf7b4b9d4797992877fa0c78fb78 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:20:23.445451 29494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 798abf7b4b9d4797992877fa0c78fb78 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:20:23.449558 29468 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 798abf7b4b9d4797992877fa0c78fb78 [term 1 FOLLOWER]: Refusing update from remote peer 453738e3a26047c5a01f41f8531cb803: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:20:23.450642 29577 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803 [LEADER]: Connected to new peer: Peer: permanent_uuid: "798abf7b4b9d4797992877fa0c78fb78" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:20:23.452971 29469 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 798abf7b4b9d4797992877fa0c78fb78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453738e3a26047c5a01f41f8531cb803 in term 1.
I20260926 15:20:23.458705 29620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798abf7b4b9d4797992877fa0c78fb78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "453738e3a26047c5a01f41f8531cb803" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38959 } } peers { permanent_uuid: "798abf7b4b9d4797992877fa0c78fb78" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } } peers { permanent_uuid: "44bedcd77f034aa6b8153ec56f2dbfd2" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } } }
I20260926 15:20:23.459649 29620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798abf7b4b9d4797992877fa0c78fb78 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:20:23.487027 29624 mvcc.cc:205] Tried to move back new op lower bound from 7333625951453646848 to 7333625951229132800. Current Snapshot: MvccSnapshot[applied={T|T < 7333625951453646848}]
I20260926 15:20:23.497895 29620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798abf7b4b9d4797992877fa0c78fb78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 453738e3a26047c5a01f41f8531cb803. Latest consensus state: current_term: 1 leader_uuid: "453738e3a26047c5a01f41f8531cb803" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "453738e3a26047c5a01f41f8531cb803" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38959 } } peers { permanent_uuid: "798abf7b4b9d4797992877fa0c78fb78" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } } peers { permanent_uuid: "44bedcd77f034aa6b8153ec56f2dbfd2" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } } }
I20260926 15:20:23.498176 29620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798abf7b4b9d4797992877fa0c78fb78 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:20:23.498430 29622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798abf7b4b9d4797992877fa0c78fb78 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "453738e3a26047c5a01f41f8531cb803" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "453738e3a26047c5a01f41f8531cb803" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38959 } } peers { permanent_uuid: "798abf7b4b9d4797992877fa0c78fb78" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } } peers { permanent_uuid: "44bedcd77f034aa6b8153ec56f2dbfd2" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } } }
I20260926 15:20:23.498835 29622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798abf7b4b9d4797992877fa0c78fb78 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 15:20:23.510705 29498 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:23.500388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46928 (local address 127.6.209.253:42265)
0926 15:20:23.500583 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:23.500589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:23.506046 (+ 5457us) server_negotiation.cc:407] Connection header received
0926 15:20:23.508945 (+ 2899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:23.508952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:23.509064 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:23.509141 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:23.510423 (+ 1282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46928: BlockingRecv error: recv got EOF from 127.0.0.1:46928 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 15:20:23.512035 29423 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:23.491608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55364 (local address 127.6.209.254:38959)
0926 15:20:23.491843 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:23.491849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:23.507760 (+ 15911us) server_negotiation.cc:407] Connection header received
0926 15:20:23.508189 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:23.508197 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:23.508313 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:23.508384 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:23.511773 (+ 3389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55364: BlockingRecv error: recv got EOF from 127.0.0.1:55364 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 15:20:23.515054 29573 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:23.506679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41780 (local address 127.6.209.252:42241)
0926 15:20:23.507086 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:23.507094 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:23.513005 (+ 5911us) server_negotiation.cc:407] Connection header received
0926 15:20:23.513520 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:23.513528 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:23.513627 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:23.513714 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:23.514769 (+ 1055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41780: BlockingRecv error: recv got EOF from 127.0.0.1:41780 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 15:20:23.543092 29646 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260926 15:20:23.548746 29498 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:23.546653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46944 (local address 127.6.209.253:42265)
0926 15:20:23.546820 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:23.546829 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:23.547297 (+ 468us) server_negotiation.cc:407] Connection header received
0926 15:20:23.547523 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:23.547529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:23.547631 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:23.547718 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:23.548490 (+ 772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46944: BlockingRecv error: recv got EOF from 127.0.0.1:46944 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 15:20:23.562424 29498 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:23.558198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46958 (local address 127.6.209.253:42265)
0926 15:20:23.558365 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:23.558373 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:23.561328 (+ 2955us) server_negotiation.cc:407] Connection header received
0926 15:20:23.561494 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:23.561498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:23.561559 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:23.561640 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:23.562259 (+ 619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46958: BlockingRecv error: recv got EOF from 127.0.0.1:46958 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 15:20:23.575903 29498 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:23.573806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46962 (local address 127.6.209.253:42265)
0926 15:20:23.573970 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:23.573974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:23.574617 (+ 643us) server_negotiation.cc:407] Connection header received
0926 15:20:23.574794 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:23.574798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:23.574876 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:23.574962 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:23.575714 (+ 752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46962: BlockingRecv error: recv got EOF from 127.0.0.1:46962 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 15:20:23.584193 29498 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:23.582484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46968 (local address 127.6.209.253:42265)
0926 15:20:23.582682 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:23.582689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:23.583121 (+ 432us) server_negotiation.cc:407] Connection header received
0926 15:20:23.583294 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:23.583299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:23.583361 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:23.583449 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:23.584026 (+ 577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46968: BlockingRecv error: recv got EOF from 127.0.0.1:46968 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260926 15:20:23.596190 29498 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:23.594437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46982 (local address 127.6.209.253:42265)
0926 15:20:23.594617 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:23.594625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:23.595077 (+ 452us) server_negotiation.cc:407] Connection header received
0926 15:20:23.595236 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:23.595241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:23.595302 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:23.595392 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:23.595966 (+ 574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46982: BlockingRecv error: recv got EOF from 127.0.0.1:46982 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 15:20:23.616118 29498 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:23.614443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46986 (local address 127.6.209.253:42265)
0926 15:20:23.614593 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:23.614600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:23.615032 (+ 432us) server_negotiation.cc:407] Connection header received
0926 15:20:23.615196 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:23.615200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:23.615269 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:23.615357 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:23.615943 (+ 586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46986: BlockingRecv error: recv got EOF from 127.0.0.1:46986 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 15:20:23.656929 29498 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:23.654713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47000 (local address 127.6.209.253:42265)
0926 15:20:23.654889 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:23.654894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:23.655417 (+ 523us) server_negotiation.cc:407] Connection header received
0926 15:20:23.655655 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:23.655662 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:23.655746 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:23.655861 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:23.656660 (+ 799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47000: BlockingRecv error: recv got EOF from 127.0.0.1:47000 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 15:20:23.726589 29498 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:23.724265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47016 (local address 127.6.209.253:42265)
0926 15:20:23.724466 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:23.724473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:23.725059 (+ 586us) server_negotiation.cc:407] Connection header received
0926 15:20:23.725271 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:23.725277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:23.725371 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:23.725481 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:23.726357 (+ 876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47016: BlockingRecv error: recv got EOF from 127.0.0.1:47016 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 15:20:23.860747 29498 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:23.858764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47030 (local address 127.6.209.253:42265)
0926 15:20:23.858982 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:23.858990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:23.859312 (+ 322us) server_negotiation.cc:407] Connection header received
0926 15:20:23.859590 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:23.859595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:23.859666 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:23.859775 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:23.860546 (+ 771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47030: BlockingRecv error: recv got EOF from 127.0.0.1:47030 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 15:20:23.901507 29577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "453738e3a26047c5a01f41f8531cb803" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "453738e3a26047c5a01f41f8531cb803" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 38959 } } peers { permanent_uuid: "798abf7b4b9d4797992877fa0c78fb78" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 42265 } } peers { permanent_uuid: "44bedcd77f034aa6b8153ec56f2dbfd2" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 42241 } } }
I20260926 15:20:23.901849 29577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 453738e3a26047c5a01f41f8531cb803 [sys.catalog]: This master's current role is: LEADER
W20260926 15:20:24.121997 29498 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:24.120171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47040 (local address 127.6.209.253:42265)
0926 15:20:24.120386 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:24.120392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:24.120642 (+ 250us) server_negotiation.cc:407] Connection header received
0926 15:20:24.120898 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:24.120902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:24.120970 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:24.121053 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:24.121802 (+ 749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47040: BlockingRecv error: recv got EOF from 127.0.0.1:47040 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 15:20:24.261086 29606 catalog_manager.cc:1259] Loaded cluster ID: 4cd8dc8e79f84936adf50a0858596341
I20260926 15:20:24.261221 29606 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 44bedcd77f034aa6b8153ec56f2dbfd2: loading cluster ID for follower catalog manager: success
I20260926 15:20:24.263435 29606 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 44bedcd77f034aa6b8153ec56f2dbfd2: acquiring CA information for follower catalog manager: success
I20260926 15:20:24.265102 29606 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 44bedcd77f034aa6b8153ec56f2dbfd2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 15:20:24.547317 29646 catalog_manager.cc:1259] Loaded cluster ID: 4cd8dc8e79f84936adf50a0858596341
I20260926 15:20:24.547430 29646 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 798abf7b4b9d4797992877fa0c78fb78: loading cluster ID for follower catalog manager: success
I20260926 15:20:24.549618 29646 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 798abf7b4b9d4797992877fa0c78fb78: acquiring CA information for follower catalog manager: success
I20260926 15:20:24.551133 29646 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 798abf7b4b9d4797992877fa0c78fb78: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 15:20:24.652434 29703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:24.640872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47050 (local address 127.6.209.253:42265)
0926 15:20:24.645977 (+ 5105us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:24.645982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:24.646365 (+ 383us) server_negotiation.cc:407] Connection header received
0926 15:20:24.646871 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:24.646877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:24.647072 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:24.647204 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:20:24.647624 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:24.648914 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:24.650001 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:24.650598 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:24.650705 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:24.651119 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:20:24.651804 (+ 685us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:20:24.651937 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:24.652029 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 15:20:24.652111 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4971,"thread_start_us":4853,"threads_started":1}
I20260926 15:20:24.665479 29709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:24.658871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41792 (local address 127.6.209.252:42241)
0926 15:20:24.659724 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:24.659729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:24.659744 (+ 15us) server_negotiation.cc:407] Connection header received
0926 15:20:24.659835 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:24.659840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:24.660032 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:24.660205 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:20:24.660600 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:24.661632 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:24.663282 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:24.663802 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:24.663912 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:24.664357 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:20:24.664931 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:20:24.665055 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:24.665147 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 15:20:24.665221 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":643,"threads_started":1}
I20260926 15:20:24.691882 29703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:24.671815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47054 (local address 127.6.209.253:42265)
0926 15:20:24.671991 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:24.671999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:24.685601 (+ 13602us) server_negotiation.cc:407] Connection header received
0926 15:20:24.685805 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:24.685809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:24.685869 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:24.685925 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:20:24.686631 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:24.687774 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:24.689780 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:24.690332 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:24.690452 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:24.691190 (+ 738us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:20:24.691295 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:20:24.691351 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:24.691558 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 15:20:24.691624 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 15:20:24.692406 29716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:24.672153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55376 (local address 127.6.209.254:38959)
0926 15:20:24.684779 (+ 12626us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:24.684788 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:24.685637 (+ 849us) server_negotiation.cc:407] Connection header received
0926 15:20:24.685761 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:24.685769 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:24.685970 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:24.686112 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:20:24.686849 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:24.687920 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:24.689847 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:24.690411 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:24.690535 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:24.691233 (+ 698us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:20:24.691610 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:20:24.691688 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:24.692082 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 15:20:24.692172 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12466,"thread_start_us":13350,"threads_started":1}
I20260926 15:20:24.696422 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 29353
I20260926 15:20:24.697428 29415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:20:24.698220 29709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:24.685789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41800 (local address 127.6.209.252:42241)
0926 15:20:24.685922 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:24.685926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:24.692460 (+ 6534us) server_negotiation.cc:407] Connection header received
0926 15:20:24.692669 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:24.692674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:24.692755 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:24.692839 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:20:24.693333 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:24.694187 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:24.695903 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:24.696505 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:24.696607 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:24.697247 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:20:24.697400 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:20:24.697472 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:24.697727 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 15:20:24.697830 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 15:20:24.802371 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 29353
I20260926 15:20:24.821646 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 29427
I20260926 15:20:24.822762 29489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:20:24.925727 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 29427
I20260926 15:20:24.940872 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 29501
I20260926 15:20:24.941880 29563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:20:25.035298 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 29501
2026-09-26T15:20:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4419 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:20:25 dist-test-slave-j6q7 krb5kdc[29723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:20:25 dist-test-slave-j6q7 krb5kdc[29723](info): set up 2 sockets
Sep 26 15:20:25 dist-test-slave-j6q7 krb5kdc[29723](info): commencing operation
krb5kdc: starting...
W20260926 15:20:26.041610 6983 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/krb5kdc/test-user.keytab.
Sep 26 15:20:26 dist-test-slave-j6q7 krb5kdc[29723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:20:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:20:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
Entry for principal kudu/127.6.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab.
I20260926 15:20:26.241127 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:45533
--webserver_interface=127.6.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:45533,127.6.209.253:45055,127.6.209.252:44909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:20:26.526899 29736 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 15:20:26.527230 29736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:20:26.527350 29736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:20:26.527480 29736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:20:26.533581 29736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:20:26.533671 29736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:20:26.533775 29736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:20:26.533846 29736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:20:26.533946 29736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:20:26.542761 29736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:45533,127.6.209.253:45055,127.6.209.252:44909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.254.keytab
--principal=kudu/127.6.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.254:45533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:20:26.543853 29736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:20:26.545104 29736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:20:26.555441 29744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:26.555475 29742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:26.555395 29741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:20:26.556766 29736 server_base.cc:1034] running on GCE node
I20260926 15:20:26.557471 29736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:20:26.558615 29736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:20:26.559830 29736 hybrid_clock.cc:648] HybridClock initialized: now 1790436026559773 us; error 33 us; skew 500 ppm
I20260926 15:20:26.560134 29736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:20:26 dist-test-slave-j6q7 krb5kdc[29723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436026, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:20:26.569082 29736 init.cc:377] Logged in from keytab as kudu/127.6.209.254@KRBTEST.COM (short username kudu)
I20260926 15:20:26.570343 29736 webserver.cc:460] Webserver started at http://127.6.209.254:46433/ using document root <none> and password file <none>
I20260926 15:20:26.571081 29736 fs_manager.cc:359] Metadata directory not provided
I20260926 15:20:26.571198 29736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:20:26.571516 29736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:20:26.574569 29736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-0/data/instance:
uuid: "366534b8b9074bb0b56443a653805985"
format_stamp: "Formatted at 2026-09-26 15:20:26 on dist-test-slave-j6q7"
server_key: "a1c2683921ded8bdd334a623b9dc5131"
server_key_iv: "9f8522df31093422232dd330d7e7f216"
server_key_version: "encryptionkey@0"
I20260926 15:20:26.575302 29736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-0/wal/instance:
uuid: "366534b8b9074bb0b56443a653805985"
format_stamp: "Formatted at 2026-09-26 15:20:26 on dist-test-slave-j6q7"
server_key: "a1c2683921ded8bdd334a623b9dc5131"
server_key_iv: "9f8522df31093422232dd330d7e7f216"
server_key_version: "encryptionkey@0"
I20260926 15:20:26.580410 29736 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260926 15:20:26.583449 29751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:20:26.584540 29736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:20:26.584695 29736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-0/wal
uuid: "366534b8b9074bb0b56443a653805985"
format_stamp: "Formatted at 2026-09-26 15:20:26 on dist-test-slave-j6q7"
server_key: "a1c2683921ded8bdd334a623b9dc5131"
server_key_iv: "9f8522df31093422232dd330d7e7f216"
server_key_version: "encryptionkey@0"
I20260926 15:20:26.584955 29736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:20:26.630457 29736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:20:26.634222 29736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:20:26.634495 29736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:20:26.657552 29736 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.254:45533
I20260926 15:20:26.657557 29802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.254:45533 every 8 connection(s)
I20260926 15:20:26.658768 29736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-0/data/info.pb
I20260926 15:20:26.661048 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 29736
I20260926 15:20:26.661250 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-0/wal/instance
I20260926 15:20:26.661695 6983 external_mini_cluster.cc:1428] Setting key 8be842130bf4f297f91e8c0993f67b1b
I20260926 15:20:26.662039 29803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:20:26.667448 29803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.254" port: 45533 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:20:26 dist-test-slave-j6q7 krb5kdc[29723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436026, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.209.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.253@KRBTEST.COM" created.
I20260926 15:20:26.687276 29806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:26.670223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45288 (local address 127.6.209.254:45533)
0926 15:20:26.670903 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:26.670920 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:26.671280 (+ 360us) server_negotiation.cc:407] Connection header received
0926 15:20:26.673142 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:26.673146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:26.673341 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:26.673520 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:26.674566 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:26.675537 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:26.676724 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:26.677323 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:26.677544 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:26.680420 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:26.680431 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:26.680445 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:26.680487 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:26.683427 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:26.684142 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:26.684147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:26.684149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:26.684326 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:26.684788 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:26.684795 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:26.684801 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:26.685298 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:26.685453 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:26.686033 (+ 580us) server_negotiation.cc:299] Negotiation successful
0926 15:20:26.686285 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":475,"thread_start_us":304,"threads_started":1}
I20260926 15:20:26.687288 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:26.670246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:45533 (local address 127.0.0.1:45288)
0926 15:20:26.671152 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:26.671184 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 15:20:26.672787 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:20:26.673548 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:20:26.673553 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:20:26.674113 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:20:26.674379 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:26.674400 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:26.675673 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:26.675678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:26.676585 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:26.676596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:26.677551 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:26.677558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:26.677826 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:26.678368 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:20:26.678392 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:20:26.680165 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:20:26.683554 (+ 3389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:26.683563 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:26.683569 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:26.684009 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:26.684465 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:26.684472 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:26.684477 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:26.684678 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:26.685439 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:20:26.685448 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:20:26.685675 (+ 227us) client_negotiation.cc:769] Sending connection context
0926 15:20:26.685935 (+ 260us) client_negotiation.cc:240] Negotiation successful
0926 15:20:26.686444 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":115,"thread_start_us":587,"threads_started":1}
I20260926 15:20:26.690080 29803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:20:26.691145 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:26.690716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:45055 (local address 127.0.0.1:57234)
0926 15:20:26.690911 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:26.691023 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 15:20:26.691434 29752 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.209.253:45055: connect: Connection refused (error 111)
I20260926 15:20:26.692766 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:26.692486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:45055 (local address 127.0.0.1:57238)
0926 15:20:26.692622 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:26.692678 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 15:20:26.693012 29803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.253:45055: Network error: Client connection negotiation failed: client connection to 127.6.209.253:45055: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.253.keytab.
Entry for principal kudu/127.6.209.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.253.keytab.
I20260926 15:20:26.711126 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.253:45055
--webserver_interface=127.6.209.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.253.keytab
--principal=kudu/127.6.209.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:45533,127.6.209.253:45055,127.6.209.252:44909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:20:26.742277 29803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } attempt: 1
I20260926 15:20:26.743321 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:26.742931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:45055 (local address 127.0.0.1:57242)
0926 15:20:26.743092 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:26.743182 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 15:20:26.744194 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:26.743888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:45055 (local address 127.0.0.1:57250)
0926 15:20:26.744013 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:26.744072 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 15:20:26.744359 29803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.253:45055: Network error: Client connection negotiation failed: client connection to 127.6.209.253:45055: connect: Connection refused (error 111)
I20260926 15:20:26.812592 29803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } attempt: 2
I20260926 15:20:26.813745 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:26.813240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:45055 (local address 127.0.0.1:57256)
0926 15:20:26.813530 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:26.813624 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 15:20:26.814602 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:26.814289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:45055 (local address 127.0.0.1:57260)
0926 15:20:26.814408 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:26.814478 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 15:20:26.814756 29803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.253:45055: Network error: Client connection negotiation failed: client connection to 127.6.209.253:45055: connect: Connection refused (error 111)
I20260926 15:20:26.906049 29803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } attempt: 3
I20260926 15:20:26.907337 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:26.906729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:45055 (local address 127.0.0.1:57276)
0926 15:20:26.906991 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:26.907123 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 15:20:26.908458 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:26.908027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:45055 (local address 127.0.0.1:57280)
0926 15:20:26.908186 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:26.908286 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 15:20:26.908649 29803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.253:45055: Network error: Client connection negotiation failed: client connection to 127.6.209.253:45055: connect: Connection refused (error 111)
W20260926 15:20:27.008123 29810 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 15:20:27.008430 29810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:20:27.008543 29810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:20:27.008673 29810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:20:27.015277 29810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:20:27.015352 29810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:20:27.015406 29810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:20:27.015507 29810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:20:27.015631 29810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:20:27.024168 29810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:45533,127.6.209.253:45055,127.6.209.252:44909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.253.keytab
--principal=kudu/127.6.209.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.253:45055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.209.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:20:27.025285 29810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:20:27.026423 29810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:20:27.039500 29816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:27.039724 29818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:27.040198 29815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:20:27.040485 29810 server_base.cc:1034] running on GCE node
I20260926 15:20:27.041450 29810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:20:27.042992 29810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:20:27.044236 29810 hybrid_clock.cc:648] HybridClock initialized: now 1790436027044200 us; error 55 us; skew 500 ppm
I20260926 15:20:27.044581 29810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:20:27 dist-test-slave-j6q7 krb5kdc[29723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436027, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:20:27.048736 29810 init.cc:377] Logged in from keytab as kudu/127.6.209.253@KRBTEST.COM (short username kudu)
I20260926 15:20:27.050000 29810 webserver.cc:460] Webserver started at http://127.6.209.253:33191/ using document root <none> and password file <none>
I20260926 15:20:27.050550 29810 fs_manager.cc:359] Metadata directory not provided
I20260926 15:20:27.050643 29810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:20:27.050925 29810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:20:27.051875 29803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } attempt: 4
I20260926 15:20:27.052995 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.052446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:45055 (local address 127.0.0.1:57284)
0926 15:20:27.052713 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.052824 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260926 15:20:27.054052 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.053734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:45055 (local address 127.0.0.1:57300)
0926 15:20:27.053871 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.053949 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.253:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 15:20:27.054253 29803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.253:45055: Network error: Client connection negotiation failed: client connection to 127.6.209.253:45055: connect: Connection refused (error 111)
I20260926 15:20:27.054158 29810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-1/data/instance:
uuid: "3be54a518a6845d5bcb4030f2bbcc7c7"
format_stamp: "Formatted at 2026-09-26 15:20:27 on dist-test-slave-j6q7"
server_key: "106f634e3031a9c9820f2700f7432ac2"
server_key_iv: "dc94e9a67885b6184593a21e5d1f26dd"
server_key_version: "encryptionkey@0"
I20260926 15:20:27.055018 29810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-1/wal/instance:
uuid: "3be54a518a6845d5bcb4030f2bbcc7c7"
format_stamp: "Formatted at 2026-09-26 15:20:27 on dist-test-slave-j6q7"
server_key: "106f634e3031a9c9820f2700f7432ac2"
server_key_iv: "dc94e9a67885b6184593a21e5d1f26dd"
server_key_version: "encryptionkey@0"
I20260926 15:20:27.060645 29810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 15:20:27.064604 29825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:20:27.066200 29810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:20:27.066360 29810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-1/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-1/wal
uuid: "3be54a518a6845d5bcb4030f2bbcc7c7"
format_stamp: "Formatted at 2026-09-26 15:20:27 on dist-test-slave-j6q7"
server_key: "106f634e3031a9c9820f2700f7432ac2"
server_key_iv: "dc94e9a67885b6184593a21e5d1f26dd"
server_key_version: "encryptionkey@0"
I20260926 15:20:27.066596 29810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:20:27.086189 29810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:20:27.089785 29810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:20:27.090056 29810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:20:27.106911 29810 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.253:45055
I20260926 15:20:27.106935 29876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.253:45055 every 8 connection(s)
I20260926 15:20:27.108157 29810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-1/data/info.pb
I20260926 15:20:27.109342 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 29810
I20260926 15:20:27.109517 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-1/wal/instance
I20260926 15:20:27.109992 6983 external_mini_cluster.cc:1428] Setting key 3a4549641a1b83e3a8250d2add6900e8
I20260926 15:20:27.112071 29877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:20:27.120724 29877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.254" port: 45533 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:20:27 dist-test-slave-j6q7 krb5kdc[29723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436027, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.253@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.209.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.209.252@KRBTEST.COM" created.
I20260926 15:20:27.139230 29806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.122959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45298 (local address 127.6.209.254:45533)
0926 15:20:27.123101 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:27.123105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:27.124244 (+ 1139us) server_negotiation.cc:407] Connection header received
0926 15:20:27.125856 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:27.125861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:27.125955 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:27.126051 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:27.127355 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.128011 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.129367 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.129802 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.129948 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:27.133936 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:27.133940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:27.133943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:27.133971 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:27.136255 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.137103 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.137108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.137110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.137228 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.137670 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.137673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.137675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.137949 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:27.138055 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:27.138853 (+ 798us) server_negotiation.cc:299] Negotiation successful
0926 15:20:27.139006 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 15:20:27.140590 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.123503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:45533 (local address 127.0.0.1:45298)
0926 15:20:27.124107 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.124142 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 15:20:27.125613 (+ 1471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:20:27.126240 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:20:27.126247 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:20:27.126698 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:20:27.127185 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:27.127201 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.128150 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.128154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:27.129190 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:27.129201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.130029 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.130035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:27.130247 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:27.130881 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:20:27.130918 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:20:27.133747 (+ 2829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:20:27.136395 (+ 2648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:27.136403 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:27.136416 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:27.136971 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:27.137336 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:27.137343 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:27.137350 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:27.137562 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:27.138078 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:20:27.138088 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:20:27.138505 (+ 417us) client_negotiation.cc:769] Sending connection context
0926 15:20:27.138779 (+ 274us) client_negotiation.cc:240] Negotiation successful
0926 15:20:27.139268 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":258,"threads_started":1}
I20260926 15:20:27.142131 29877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:20:27 dist-test-slave-j6q7 krb5kdc[29723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436027, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.253@KRBTEST.COM for kudu/127.6.209.253@KRBTEST.COM
I20260926 15:20:27.155642 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.142798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:45055 (local address 127.0.0.1:57308)
0926 15:20:27.142962 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.142976 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:20:27.143062 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:20:27.144012 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:20:27.144017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:20:27.144438 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:20:27.144604 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:27.144614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.145835 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.145842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:27.147378 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:27.147393 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.148122 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.148129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:27.148326 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:27.148847 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:20:27.148871 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:20:27.151037 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:20:27.153660 (+ 2623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:27.153665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:27.153669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:27.153966 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:27.154280 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:27.154284 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:27.154287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:27.154403 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:27.154858 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:20:27.154862 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:20:27.154987 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 15:20:27.155114 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 15:20:27.155320 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 15:20:27.155709 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.142962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57308 (local address 127.6.209.253:45055)
0926 15:20:27.143574 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:27.143583 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:27.143610 (+ 27us) server_negotiation.cc:407] Connection header received
0926 15:20:27.143656 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:27.143665 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:27.143852 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:27.143994 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:27.144751 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.145711 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.147508 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.147988 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.148069 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:27.151194 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:27.151202 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:27.151212 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:27.151234 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:27.153535 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.154074 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.154082 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.154086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.154182 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.154488 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.154495 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.154498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.154756 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:27.154947 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:27.155234 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 15:20:27.155400 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":316,"threads_started":1}
I20260926 15:20:27.157644 29877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:20:27.158752 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.158412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:44909 (local address 127.0.0.1:50384)
0926 15:20:27.158542 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.158637 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 15:20:27.158981 29827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
I20260926 15:20:27.160308 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.159951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:44909 (local address 127.0.0.1:50386)
0926 15:20:27.160156 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.160224 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 15:20:27.160495 29877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.252:44909: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.209.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.252.keytab.
Entry for principal kudu/127.6.209.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.252.keytab.
I20260926 15:20:27.164245 6983 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
/tmp/dist-test-taskFXn57Q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.252:44909
--webserver_interface=127.6.209.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.252.keytab
--principal=kudu/127.6.209.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.209.212:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.209.254:45533,127.6.209.253:45055,127.6.209.252:44909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:20:27.209833 29877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } attempt: 1
I20260926 15:20:27.210780 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.210405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:44909 (local address 127.0.0.1:50398)
0926 15:20:27.210578 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.210660 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 15:20:27.211632 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.211351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:44909 (local address 127.0.0.1:50414)
0926 15:20:27.211489 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.211547 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 15:20:27.211846 29877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.252:44909: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
I20260926 15:20:27.280094 29877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } attempt: 2
I20260926 15:20:27.281155 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.280731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:44909 (local address 127.0.0.1:50430)
0926 15:20:27.280923 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.281014 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 15:20:27.282047 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.281761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:44909 (local address 127.0.0.1:50440)
0926 15:20:27.281885 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.281950 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 15:20:27.282249 29877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.252:44909: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
I20260926 15:20:27.353452 29803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } attempt: 5
Sep 26 15:20:27 dist-test-slave-j6q7 krb5kdc[29723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436026, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for kudu/127.6.209.253@KRBTEST.COM
I20260926 15:20:27.365247 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.354215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57324 (local address 127.6.209.253:45055)
0926 15:20:27.354391 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:27.354398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:27.354452 (+ 54us) server_negotiation.cc:407] Connection header received
0926 15:20:27.354652 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:27.354656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:27.354726 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:27.354823 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:27.355517 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.356112 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.357299 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.357717 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.357862 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:27.360744 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:27.360749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:27.360752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:27.360781 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:27.363080 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.363616 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.363622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.363625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.363737 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.364065 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.364070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.364073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.364304 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:27.364435 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:27.364763 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 15:20:27.364972 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 15:20:27.365247 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.354136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:45055 (local address 127.0.0.1:57324)
0926 15:20:27.354348 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.354368 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:20:27.354515 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:20:27.354856 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:20:27.354861 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:20:27.355222 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:20:27.355393 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:27.355400 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.356231 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.356237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:27.357163 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:27.357174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.357878 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.357883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:27.358065 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:27.358644 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:20:27.358660 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:20:27.360565 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:20:27.363198 (+ 2633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:27.363202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:27.363206 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:27.363517 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:27.363837 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:27.363841 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:27.363845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:27.363966 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:27.364413 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:20:27.364417 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:20:27.364559 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 15:20:27.364679 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 15:20:27.364952 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 15:20:27.366328 29803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:20:27.367437 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.367091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:44909 (local address 127.0.0.1:50454)
0926 15:20:27.367235 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.367302 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 15:20:27.368250 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.367992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:44909 (local address 127.0.0.1:50464)
0926 15:20:27.368104 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.368176 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 15:20:27.368436 29803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.252:44909: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
I20260926 15:20:27.373405 29877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } attempt: 3
I20260926 15:20:27.374151 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.373840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:44909 (local address 127.0.0.1:50470)
0926 15:20:27.373984 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.374050 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 15:20:27.374986 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.374650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:44909 (local address 127.0.0.1:50474)
0926 15:20:27.374832 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.374895 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260926 15:20:27.375159 29877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.252:44909: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
I20260926 15:20:27.419708 29803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } attempt: 1
I20260926 15:20:27.420627 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.420255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:44909 (local address 127.0.0.1:50488)
0926 15:20:27.420419 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.420496 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 15:20:27.421537 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.421274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:44909 (local address 127.0.0.1:50500)
0926 15:20:27.421405 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.421459 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 15:20:27.421711 29803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.252:44909: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
W20260926 15:20:27.453819 29884 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 15:20:27.454118 29884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:20:27.454221 29884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:20:27.454331 29884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:20:27.460458 29884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:20:27.460563 29884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:20:27.460683 29884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:20:27.460764 29884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:20:27.460867 29884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:20:27.468940 29884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.209.212:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.209.254:45533,127.6.209.253:45055,127.6.209.252:44909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/krb5kdc/kudu_127.6.209.252.keytab
--principal=kudu/127.6.209.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.209.252:44909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.209.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:20:27.469915 29884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:20:27.471040 29884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:20:27.481442 29890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:27.481489 29889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:20:27.481406 29892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:20:27.482036 29884 server_base.cc:1034] running on GCE node
I20260926 15:20:27.483454 29884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:20:27.484612 29884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:20:27.485834 29884 hybrid_clock.cc:648] HybridClock initialized: now 1790436027485773 us; error 54 us; skew 500 ppm
I20260926 15:20:27.486117 29884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:20:27 dist-test-slave-j6q7 krb5kdc[29723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436027, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:20:27.489063 29884 init.cc:377] Logged in from keytab as kudu/127.6.209.252@KRBTEST.COM (short username kudu)
I20260926 15:20:27.489876 29803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } attempt: 2
I20260926 15:20:27.490314 29884 webserver.cc:460] Webserver started at http://127.6.209.252:45409/ using document root <none> and password file <none>
I20260926 15:20:27.490770 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.490400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:44909 (local address 127.0.0.1:50514)
0926 15:20:27.490571 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.490638 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 15:20:27.490967 29884 fs_manager.cc:359] Metadata directory not provided
I20260926 15:20:27.491093 29884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:20:27.491436 29884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:20:27.491714 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.491417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:44909 (local address 127.0.0.1:50522)
0926 15:20:27.491567 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.491626 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 15:20:27.491969 29803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.252:44909: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
I20260926 15:20:27.494051 29884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-2/data/instance:
uuid: "4ceb7c602e5641e9808e233ac671148e"
format_stamp: "Formatted at 2026-09-26 15:20:27 on dist-test-slave-j6q7"
server_key: "461679ea0e61a2adb457c85ac9d3ecad"
server_key_iv: "55b69631488d9e0efb6e225787fa281b"
server_key_version: "encryptionkey@0"
I20260926 15:20:27.494726 29884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-2/wal/instance:
uuid: "4ceb7c602e5641e9808e233ac671148e"
format_stamp: "Formatted at 2026-09-26 15:20:27 on dist-test-slave-j6q7"
server_key: "461679ea0e61a2adb457c85ac9d3ecad"
server_key_iv: "55b69631488d9e0efb6e225787fa281b"
server_key_version: "encryptionkey@0"
I20260926 15:20:27.499410 29884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 15:20:27.502471 29899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:20:27.503556 29884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 15:20:27.503718 29884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-2/data,/tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-2/wal
uuid: "4ceb7c602e5641e9808e233ac671148e"
format_stamp: "Formatted at 2026-09-26 15:20:27 on dist-test-slave-j6q7"
server_key: "461679ea0e61a2adb457c85ac9d3ecad"
server_key_iv: "55b69631488d9e0efb6e225787fa281b"
server_key_version: "encryptionkey@0"
I20260926 15:20:27.503926 29884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:20:27.518339 29877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } attempt: 4
I20260926 15:20:27.519177 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.518802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:44909 (local address 127.0.0.1:50538)
0926 15:20:27.518985 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.519054 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 15:20:27.520028 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.519728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:44909 (local address 127.0.0.1:50554)
0926 15:20:27.519855 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.519918 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 15:20:27.520231 29877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.209.252:44909: Network error: Client connection negotiation failed: client connection to 127.6.209.252:44909: connect: Connection refused (error 111)
I20260926 15:20:27.526829 29884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:20:27.530490 29884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:20:27.530733 29884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:20:27.546768 29884 rpc_server.cc:307] RPC server started. Bound to: 127.6.209.252:44909
I20260926 15:20:27.546792 29950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.209.252:44909 every 8 connection(s)
I20260926 15:20:27.547941 29884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-2/data/info.pb
I20260926 15:20:27.550977 29951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:20:27.553169 6983 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu as pid 29884
I20260926 15:20:27.553326 6983 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXn57Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435694426955-6983-0/minicluster-data/master-2/wal/instance
I20260926 15:20:27.553793 6983 external_mini_cluster.cc:1428] Setting key 6c3c53c0244b88879e7de270e3f9c687
I20260926 15:20:27.557981 29951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.254" port: 45533 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:20:27 dist-test-slave-j6q7 krb5kdc[29723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.252@KRBTEST.COM
Sep 26 15:20:27 dist-test-slave-j6q7 krb5kdc[29723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436027, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.252@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:20:27.573804 29954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.555275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50564 (local address 127.6.209.252:44909)
0926 15:20:27.555962 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:27.555973 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:27.556009 (+ 36us) server_negotiation.cc:407] Connection header received
0926 15:20:27.557606 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:27.557626 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:27.557990 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:27.558253 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:27.559789 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.561321 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.562535 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.563043 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.563204 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:27.566594 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:27.566600 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:27.566625 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:27.566674 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:27.569762 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.570441 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.570445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.570450 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.570566 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.571151 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.571156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.571159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.571591 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:27.571865 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:27.572337 (+ 472us) server_negotiation.cc:299] Negotiation successful
0926 15:20:27.572635 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":331,"threads_started":1}
I20260926 15:20:27.574076 29955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.560514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.254:45533 (local address 127.0.0.1:45306)
0926 15:20:27.561314 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.561351 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 15:20:27.561529 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:20:27.561864 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:20:27.561867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:20:27.562319 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:20:27.562517 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:27.562524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.563356 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.563360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:27.564318 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:27.564328 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.565051 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.565058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:27.565192 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:27.565678 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:20:27.565695 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:20:27.567829 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:20:27.571178 (+ 3349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:27.571187 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:27.571191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:27.571791 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:27.572273 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:27.572278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:27.572281 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:27.572506 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:27.573207 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:20:27.573212 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:20:27.573370 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 15:20:27.573507 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 15:20:27.573691 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"spinlock_wait_cycles":16768,"thread_start_us":416,"threads_started":1}
I20260926 15:20:27.574146 29806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.560473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45306 (local address 127.6.209.254:45533)
0926 15:20:27.560680 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:27.560686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:27.561431 (+ 745us) server_negotiation.cc:407] Connection header received
0926 15:20:27.561657 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:27.561663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:27.561733 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:27.561824 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:27.562646 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.563224 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.564446 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.564844 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.564974 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:27.568035 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:27.568038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:27.568041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:27.568069 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:27.570973 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.571947 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.571950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.571953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.572114 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.572754 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.572759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.572761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.573097 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:27.573247 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:27.573623 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 15:20:27.573827 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 15:20:27.575462 29951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:20:27 dist-test-slave-j6q7 krb5kdc[29723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436027, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.252@KRBTEST.COM for kudu/127.6.209.253@KRBTEST.COM
Sep 26 15:20:27 dist-test-slave-j6q7 krb5kdc[29723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.254@KRBTEST.COM
I20260926 15:20:27.587786 29955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.576148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.253:45055 (local address 127.0.0.1:57340)
0926 15:20:27.576273 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.576285 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:20:27.576349 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:20:27.576972 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:20:27.576976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:20:27.577404 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:20:27.577543 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:27.577550 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.578419 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.578426 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:27.579146 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:27.579154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.579745 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.579750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:27.579948 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:27.580474 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:20:27.580489 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:20:27.582392 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:20:27.585688 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:27.585691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:27.585693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:27.585988 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:27.586331 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:27.586333 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:27.586335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:27.586439 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:27.586952 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:20:27.586957 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:20:27.587108 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 15:20:27.587237 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 15:20:27.587468 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 15:20:27.587919 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.576313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57340 (local address 127.6.209.253:45055)
0926 15:20:27.576515 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:27.576521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:27.576540 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:20:27.576595 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:27.576600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:27.576691 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:27.576788 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:27.577686 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.578302 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.579270 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.579619 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.579736 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:27.582566 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:27.582574 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:27.582579 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:27.582621 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:27.585578 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.586092 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.586097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.586101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.586233 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.586527 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.586531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.586533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.586831 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:27.586988 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:27.587346 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 15:20:27.587548 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 15:20:27.588735 29951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:20:27.589147 29806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.576741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.6.209.254:45533)
0926 15:20:27.576888 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:27.576893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:27.577039 (+ 146us) server_negotiation.cc:407] Connection header received
0926 15:20:27.577124 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:27.577131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:27.577202 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:27.577312 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:27.578244 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.578836 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.579816 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.580254 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.580375 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:27.583883 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:27.583890 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:27.583896 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:27.583934 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:27.586568 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.587142 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.587148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.587152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.587276 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.587642 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.587646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.587650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.588005 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:27.588296 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:27.588625 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 15:20:27.588790 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Sep 26 15:20:27 dist-test-slave-j6q7 krb5kdc[29723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436027, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.252@KRBTEST.COM for kudu/127.6.209.252@KRBTEST.COM
I20260926 15:20:27.598186 29803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } attempt: 3
Sep 26 15:20:27 dist-test-slave-j6q7 krb5kdc[29723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.209.253@KRBTEST.COM
I20260926 15:20:27.602283 29955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.589348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:44909 (local address 127.0.0.1:50566)
0926 15:20:27.589460 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.589472 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:20:27.589571 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:20:27.589946 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:20:27.589950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:20:27.590284 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:20:27.590446 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:27.590457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.591246 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.591250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:27.592416 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:27.592425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.593185 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.593188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:27.593413 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:27.593843 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:20:27.593860 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:20:27.596195 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:20:27.599632 (+ 3437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:27.599638 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:27.599643 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:27.600133 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:27.600525 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:27.600533 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:27.600539 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:27.600728 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:27.601351 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:20:27.601360 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:20:27.601508 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 15:20:27.601657 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 15:20:27.601931 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 15:20:27.602901 29954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.589620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50566 (local address 127.6.209.252:44909)
0926 15:20:27.589731 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:27.589737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:27.589755 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:20:27.589786 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:27.589788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:27.589828 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:27.589894 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:27.590565 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.591156 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.592521 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.593076 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.593179 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:27.596348 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:27.596355 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:27.596361 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:27.596395 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:27.599514 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.600254 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.600259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.600262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.600406 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.600869 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.600876 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.600880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.601202 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:27.601583 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:27.601731 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 15:20:27.602600 (+ 869us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":681,"server-negotiator.queue_time_us":27}
I20260926 15:20:27.605161 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.590568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57352 (local address 127.6.209.253:45055)
0926 15:20:27.590713 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:27.590720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:27.590765 (+ 45us) server_negotiation.cc:407] Connection header received
0926 15:20:27.590924 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:27.590928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:27.590985 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:27.591066 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:27.592281 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.593162 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.594934 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.595320 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.595403 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:27.600011 (+ 4608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:27.600017 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:27.600021 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:27.600074 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:27.602977 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.603560 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.603566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.603569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.603698 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.604035 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.604040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.604043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.604247 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:27.604477 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:27.604714 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 15:20:27.604935 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:20:27.607987 29951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ceb7c602e5641e9808e233ac671148e: Bootstrap starting.
I20260926 15:20:27.608989 6983 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:20:27.610909 29951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ceb7c602e5641e9808e233ac671148e: Neither blocks nor log segments found. Creating new log.
I20260926 15:20:27.612141 29951 log.cc:824] T 00000000000000000000000000000000 P 4ceb7c602e5641e9808e233ac671148e: Log is configured to *not* fsync() on all Append() calls
Sep 26 15:20:27 dist-test-slave-j6q7 krb5kdc[29723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436026, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.254@KRBTEST.COM for kudu/127.6.209.252@KRBTEST.COM
I20260926 15:20:27.610924 29778 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "366534b8b9074bb0b56443a653805985"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45318
I20260926 15:20:27.615247 29951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ceb7c602e5641e9808e233ac671148e: No bootstrap required, opened a new log
I20260926 15:20:27.618820 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.598634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:44909 (local address 127.0.0.1:50572)
0926 15:20:27.598845 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.598864 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:20:27.598984 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:20:27.599713 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:20:27.599718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:20:27.600150 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:20:27.600356 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:27.600367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.601949 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.601954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:27.602972 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:27.602988 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.603819 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.603824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:27.603993 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:27.611144 (+ 7151us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:20:27.611159 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:20:27.613542 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:20:27.616379 (+ 2837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:27.616384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:27.616387 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:27.616755 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:27.617196 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:27.617201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:27.617205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:27.617343 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:27.617919 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:20:27.617924 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:20:27.618097 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 15:20:27.618229 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 15:20:27.618469 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 15:20:27.618906 29956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.598667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50572 (local address 127.6.209.252:44909)
0926 15:20:27.599483 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:27.599486 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:27.599500 (+ 14us) server_negotiation.cc:407] Connection header received
0926 15:20:27.599529 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:27.599531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:27.599569 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:27.599679 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:27.600545 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.601828 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.603125 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.603683 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.603800 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:27.613720 (+ 9920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:27.613725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:27.613729 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:27.613762 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:27.616256 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.616920 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.616927 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.616931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.617067 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.617488 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.617495 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.617498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.617756 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:27.617951 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:27.618321 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 15:20:27.618514 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":595,"threads_started":1}
I20260926 15:20:27.623203 29951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ceb7c602e5641e9808e233ac671148e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366534b8b9074bb0b56443a653805985" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 45533 } } peers { permanent_uuid: "3be54a518a6845d5bcb4030f2bbcc7c7" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } } peers { permanent_uuid: "4ceb7c602e5641e9808e233ac671148e" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } }
I20260926 15:20:27.623484 29951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ceb7c602e5641e9808e233ac671148e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ceb7c602e5641e9808e233ac671148e, State: Initialized, Role: FOLLOWER
I20260926 15:20:27.623642 29803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985: Bootstrap starting.
I20260926 15:20:27.623905 29951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ceb7c602e5641e9808e233ac671148e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366534b8b9074bb0b56443a653805985" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 45533 } } peers { permanent_uuid: "3be54a518a6845d5bcb4030f2bbcc7c7" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } } peers { permanent_uuid: "4ceb7c602e5641e9808e233ac671148e" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } }
I20260926 15:20:27.624640 29958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ceb7c602e5641e9808e233ac671148e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366534b8b9074bb0b56443a653805985" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 45533 } } peers { permanent_uuid: "3be54a518a6845d5bcb4030f2bbcc7c7" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } } peers { permanent_uuid: "4ceb7c602e5641e9808e233ac671148e" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } } }
I20260926 15:20:27.624989 29958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ceb7c602e5641e9808e233ac671148e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:20:27.625658 29951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ceb7c602e5641e9808e233ac671148e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:20:27.627051 29803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985: Neither blocks nor log segments found. Creating new log.
I20260926 15:20:27.628171 29803 log.cc:824] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985: Log is configured to *not* fsync() on all Append() calls
I20260926 15:20:27.630919 29803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985: No bootstrap required, opened a new log
I20260926 15:20:27.637688 29803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366534b8b9074bb0b56443a653805985" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 45533 } } peers { permanent_uuid: "3be54a518a6845d5bcb4030f2bbcc7c7" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } } peers { permanent_uuid: "4ceb7c602e5641e9808e233ac671148e" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } }
I20260926 15:20:27.638026 29803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 366534b8b9074bb0b56443a653805985, State: Initialized, Role: FOLLOWER
I20260926 15:20:27.638430 29803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366534b8b9074bb0b56443a653805985" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 45533 } } peers { permanent_uuid: "3be54a518a6845d5bcb4030f2bbcc7c7" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } } peers { permanent_uuid: "4ceb7c602e5641e9808e233ac671148e" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } }
W20260926 15:20:27.638979 29970 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ceb7c602e5641e9808e233ac671148e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:20:27.639096 29970 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:20:27.639320 29972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366534b8b9074bb0b56443a653805985" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 45533 } } peers { permanent_uuid: "3be54a518a6845d5bcb4030f2bbcc7c7" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } } peers { permanent_uuid: "4ceb7c602e5641e9808e233ac671148e" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } } }
I20260926 15:20:27.639673 29972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:20:27.640203 29803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:20:27.641032 29778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 15:20:27.641232 29778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:20:27.644779 29778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366534b8b9074bb0b56443a653805985" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 45533 } } peers { permanent_uuid: "3be54a518a6845d5bcb4030f2bbcc7c7" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } } peers { permanent_uuid: "4ceb7c602e5641e9808e233ac671148e" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } }
I20260926 15:20:27.636428 29884 master_runner.cc:186] Error getting master registration for 127.6.209.254:45533: OK, instance_id { permanent_uuid: "366534b8b9074bb0b56443a653805985" instance_seqno: 1790436026652801 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 15:20:27.646387 29884 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 15:20:27.647153 29778 leader_election.cc:288] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985 [CANDIDATE]: Term 1 election: Requested vote from peers 3be54a518a6845d5bcb4030f2bbcc7c7 (127.6.209.253:45055), 4ceb7c602e5641e9808e233ac671148e (127.6.209.252:44909)
I20260926 15:20:27.647564 29926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "366534b8b9074bb0b56443a653805985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4ceb7c602e5641e9808e233ac671148e"
I20260926 15:20:27.647935 29926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ceb7c602e5641e9808e233ac671148e [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:20:27.650722 29926 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4ceb7c602e5641e9808e233ac671148e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 366534b8b9074bb0b56443a653805985 in term 1.
I20260926 15:20:27.651175 29753 leader_election.cc:302] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 366534b8b9074bb0b56443a653805985, 4ceb7c602e5641e9808e233ac671148e; no voters:
I20260926 15:20:27.651566 29972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:20:27.652204 29972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985 [term 1 LEADER]: Becoming Leader. State: Replica: 366534b8b9074bb0b56443a653805985, State: Running, Role: LEADER
I20260926 15:20:27.652774 29972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366534b8b9074bb0b56443a653805985" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 45533 } } peers { permanent_uuid: "3be54a518a6845d5bcb4030f2bbcc7c7" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } } peers { permanent_uuid: "4ceb7c602e5641e9808e233ac671148e" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } }
I20260926 15:20:27.672925 29977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 366534b8b9074bb0b56443a653805985. Latest consensus state: current_term: 1 leader_uuid: "366534b8b9074bb0b56443a653805985" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366534b8b9074bb0b56443a653805985" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 45533 } } peers { permanent_uuid: "3be54a518a6845d5bcb4030f2bbcc7c7" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } } peers { permanent_uuid: "4ceb7c602e5641e9808e233ac671148e" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } } }
I20260926 15:20:27.673409 29977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985 [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:27.681504 29979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:20:27.690685 29979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:20:27.728370 29926 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4ceb7c602e5641e9808e233ac671148e [term 1 FOLLOWER]: Refusing update from remote peer 366534b8b9074bb0b56443a653805985: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:20:27.729806 29977 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ceb7c602e5641e9808e233ac671148e" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:20:27.740229 29806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.687393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45322 (local address 127.6.209.254:45533)
0926 15:20:27.687756 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:27.687762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:27.694523 (+ 6761us) server_negotiation.cc:407] Connection header received
0926 15:20:27.695001 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:27.695009 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:27.695103 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:27.695181 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:27.700157 (+ 4976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.701628 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.705204 (+ 3576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.706097 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.706224 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:27.726764 (+ 20540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:27.726771 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:27.726776 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:27.726832 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:27.730308 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.735496 (+ 5188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.735501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.735505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.735649 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.737113 (+ 1464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.737118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.737121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.737487 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:27.737742 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:27.739590 (+ 1848us) server_negotiation.cc:299] Negotiation successful
0926 15:20:27.739802 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253}
I20260926 15:20:27.739933 29958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ceb7c602e5641e9808e233ac671148e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 366534b8b9074bb0b56443a653805985. Latest consensus state: current_term: 1 leader_uuid: "366534b8b9074bb0b56443a653805985" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366534b8b9074bb0b56443a653805985" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 45533 } } peers { permanent_uuid: "3be54a518a6845d5bcb4030f2bbcc7c7" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } } peers { permanent_uuid: "4ceb7c602e5641e9808e233ac671148e" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } } }
I20260926 15:20:27.740329 29958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ceb7c602e5641e9808e233ac671148e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:20:27.744956 29987 mvcc.cc:205] Tried to move back new op lower bound from 7333625969527742464 to 7333625969271267328. Current Snapshot: MvccSnapshot[applied={T|T < 7333625969527742464}]
I20260926 15:20:27.749509 29958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ceb7c602e5641e9808e233ac671148e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "366534b8b9074bb0b56443a653805985" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366534b8b9074bb0b56443a653805985" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 45533 } } peers { permanent_uuid: "3be54a518a6845d5bcb4030f2bbcc7c7" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } } peers { permanent_uuid: "4ceb7c602e5641e9808e233ac671148e" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } } }
I20260926 15:20:27.749831 29958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ceb7c602e5641e9808e233ac671148e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:20:27.758984 29956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.686363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50582 (local address 127.6.209.252:44909)
0926 15:20:27.686535 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:27.686540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:27.710459 (+ 23919us) server_negotiation.cc:407] Connection header received
0926 15:20:27.710768 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:27.710774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:27.710888 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:27.710956 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:27.716833 (+ 5877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.717959 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.731141 (+ 13182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.731854 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.731937 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:27.745736 (+ 13799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:27.745742 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:27.745747 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:27.745783 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:27.749638 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.754506 (+ 4868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.754511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.754514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.754652 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.755574 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.755579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.755582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.755879 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:27.756070 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:27.758483 (+ 2413us) server_negotiation.cc:299] Negotiation successful
0926 15:20:27.758675 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 15:20:27.762946 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.687581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57362 (local address 127.6.209.253:45055)
0926 15:20:27.687909 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:27.687914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:27.712261 (+ 24347us) server_negotiation.cc:407] Connection header received
0926 15:20:27.712872 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:27.712878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:27.712952 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:27.713018 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:27.720116 (+ 7098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.721245 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.733867 (+ 12622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.734432 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.734518 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:27.750732 (+ 16214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:27.750737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:27.750742 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:27.750786 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:27.753948 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.756434 (+ 2486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.756439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.756442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.756576 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.757656 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.757662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.757665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.758034 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:27.758216 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:27.762398 (+ 4182us) server_negotiation.cc:299] Negotiation successful
0926 15:20:27.762621 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260926 15:20:27.763490 29972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "366534b8b9074bb0b56443a653805985" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366534b8b9074bb0b56443a653805985" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 45533 } } peers { permanent_uuid: "3be54a518a6845d5bcb4030f2bbcc7c7" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } } peers { permanent_uuid: "4ceb7c602e5641e9808e233ac671148e" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } } }
I20260926 15:20:27.763857 29972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985 [sys.catalog]: This master's current role is: LEADER
I20260926 15:20:27.768481 29979 catalog_manager.cc:1347] Generated new cluster ID: 9f024d388b8945f888039601df7905a4
I20260926 15:20:27.768572 29979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:20:27.794050 29999 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 15:20:27.797709 29736 master_runner.cc:186] Error getting master registration for 127.6.209.253:45055: OK, instance_id { permanent_uuid: "3be54a518a6845d5bcb4030f2bbcc7c7" instance_seqno: 1790436027102255 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 15:20:27.808128 29736 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 15:20:27.819419 29877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } attempt: 5
Sep 26 15:20:27 dist-test-slave-j6q7 krb5kdc[29723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790436027, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.209.253@KRBTEST.COM for kudu/127.6.209.252@KRBTEST.COM
I20260926 15:20:27.837757 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.819916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.209.252:44909 (local address 127.0.0.1:50588)
0926 15:20:27.820145 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 15:20:27.820166 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:20:27.820288 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:20:27.820736 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:20:27.820741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:20:27.821288 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:20:27.821518 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:27.821530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.822876 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.822882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:27.824195 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:20:27.824210 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.825082 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.825088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:20:27.825325 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:27.825926 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:20:27.825948 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:20:27.830210 (+ 4262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:20:27.834439 (+ 4229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:27.834444 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:27.834448 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:27.834943 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:27.835560 (+ 617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:20:27.835565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:20:27.835580 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:20:27.835741 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:20:27.836769 (+ 1028us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:20:27.836776 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:20:27.836953 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 15:20:27.837087 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 15:20:27.837363 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 15:20:27.840989 29956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:27.820000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50588 (local address 127.6.209.252:44909)
0926 15:20:27.820224 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:27.820230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:27.820254 (+ 24us) server_negotiation.cc:407] Connection header received
0926 15:20:27.820392 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:27.820398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:27.820467 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:27.820561 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:27.821664 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.822769 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.824329 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:27.824953 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:27.825079 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:27.830501 (+ 5422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:20:27.830508 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:20:27.830513 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:20:27.830550 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:20:27.834254 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.835126 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.835134 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.835139 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.835406 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:20:27.835946 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:20:27.835951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:20:27.835954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:20:27.836539 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:20:27.838406 (+ 1867us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:27.838663 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 15:20:27.839002 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 15:20:27.846365 29877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be54a518a6845d5bcb4030f2bbcc7c7: Bootstrap starting.
I20260926 15:20:27.850394 29877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3be54a518a6845d5bcb4030f2bbcc7c7: Neither blocks nor log segments found. Creating new log.
I20260926 15:20:27.851658 29877 log.cc:824] T 00000000000000000000000000000000 P 3be54a518a6845d5bcb4030f2bbcc7c7: Log is configured to *not* fsync() on all Append() calls
I20260926 15:20:27.855271 29979 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:20:27.858126 29979 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:20:27.867347 29877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be54a518a6845d5bcb4030f2bbcc7c7: No bootstrap required, opened a new log
I20260926 15:20:27.873955 29979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985: Generated new TSK 0
I20260926 15:20:27.875305 29979 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:20:27.886984 29877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3be54a518a6845d5bcb4030f2bbcc7c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366534b8b9074bb0b56443a653805985" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 45533 } } peers { permanent_uuid: "3be54a518a6845d5bcb4030f2bbcc7c7" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } } peers { permanent_uuid: "4ceb7c602e5641e9808e233ac671148e" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } }
I20260926 15:20:27.887434 29877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3be54a518a6845d5bcb4030f2bbcc7c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3be54a518a6845d5bcb4030f2bbcc7c7, State: Initialized, Role: FOLLOWER
I20260926 15:20:27.888039 29877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3be54a518a6845d5bcb4030f2bbcc7c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366534b8b9074bb0b56443a653805985" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 45533 } } peers { permanent_uuid: "3be54a518a6845d5bcb4030f2bbcc7c7" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } } peers { permanent_uuid: "4ceb7c602e5641e9808e233ac671148e" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } }
I20260926 15:20:27.898113 30001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be54a518a6845d5bcb4030f2bbcc7c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366534b8b9074bb0b56443a653805985" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 45533 } } peers { permanent_uuid: "3be54a518a6845d5bcb4030f2bbcc7c7" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } } peers { permanent_uuid: "4ceb7c602e5641e9808e233ac671148e" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } } }
I20260926 15:20:27.899821 29877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3be54a518a6845d5bcb4030f2bbcc7c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:20:27.899895 29852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "366534b8b9074bb0b56443a653805985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3be54a518a6845d5bcb4030f2bbcc7c7"
I20260926 15:20:27.900346 29852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3be54a518a6845d5bcb4030f2bbcc7c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:20:27.901089 30001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be54a518a6845d5bcb4030f2bbcc7c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:20:27.903584 29852 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3be54a518a6845d5bcb4030f2bbcc7c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 366534b8b9074bb0b56443a653805985 in term 1.
I20260926 15:20:27.904914 29851 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3be54a518a6845d5bcb4030f2bbcc7c7 [term 1 FOLLOWER]: Refusing update from remote peer 366534b8b9074bb0b56443a653805985: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:20:27.905673 29972 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3be54a518a6845d5bcb4030f2bbcc7c7" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:20:27.950016 30009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be54a518a6845d5bcb4030f2bbcc7c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "366534b8b9074bb0b56443a653805985" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366534b8b9074bb0b56443a653805985" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 45533 } } peers { permanent_uuid: "3be54a518a6845d5bcb4030f2bbcc7c7" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } } peers { permanent_uuid: "4ceb7c602e5641e9808e233ac671148e" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } } }
I20260926 15:20:27.950385 30009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be54a518a6845d5bcb4030f2bbcc7c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:20:27.950672 30001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be54a518a6845d5bcb4030f2bbcc7c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 366534b8b9074bb0b56443a653805985. Latest consensus state: current_term: 1 leader_uuid: "366534b8b9074bb0b56443a653805985" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366534b8b9074bb0b56443a653805985" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 45533 } } peers { permanent_uuid: "3be54a518a6845d5bcb4030f2bbcc7c7" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } } peers { permanent_uuid: "4ceb7c602e5641e9808e233ac671148e" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } } }
I20260926 15:20:27.950905 30001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be54a518a6845d5bcb4030f2bbcc7c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:20:27.988281 30022 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260926 15:20:28.006222 29806 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:27.994649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45326 (local address 127.6.209.254:45533)
0926 15:20:27.994809 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:27.994814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:28.004431 (+ 9617us) server_negotiation.cc:407] Connection header received
0926 15:20:28.004822 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:28.004828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:28.004915 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:28.004978 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:28.005994 (+ 1016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45326: BlockingRecv error: recv got EOF from 127.0.0.1:45326 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 15:20:28.008455 29881 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:27.999437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57376 (local address 127.6.209.253:45055)
0926 15:20:27.999586 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:27.999591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:28.006981 (+ 7390us) server_negotiation.cc:407] Connection header received
0926 15:20:28.007210 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:28.007215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:28.007300 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:28.007369 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:28.008247 (+ 878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57376: BlockingRecv error: recv got EOF from 127.0.0.1:57376 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 15:20:28.010417 29956 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:27.999137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50592 (local address 127.6.209.252:44909)
0926 15:20:27.999308 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:27.999313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:28.008999 (+ 9686us) server_negotiation.cc:407] Connection header received
0926 15:20:28.009202 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:28.009207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:28.009290 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:28.009359 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:28.010204 (+ 845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50592: BlockingRecv error: recv got EOF from 127.0.0.1:50592 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 15:20:28.018548 29881 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:28.015184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57378 (local address 127.6.209.253:45055)
0926 15:20:28.015350 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:28.015356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:28.017280 (+ 1924us) server_negotiation.cc:407] Connection header received
0926 15:20:28.017473 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:28.017478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:28.017596 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:28.017698 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:28.018352 (+ 654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57378: BlockingRecv error: recv got EOF from 127.0.0.1:57378 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 15:20:28.024628 29881 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:28.022965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57392 (local address 127.6.209.253:45055)
0926 15:20:28.023094 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:28.023099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:28.023555 (+ 456us) server_negotiation.cc:407] Connection header received
0926 15:20:28.023710 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:28.023715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:28.023782 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:28.023885 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:28.024484 (+ 599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57392: BlockingRecv error: recv got EOF from 127.0.0.1:57392 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 15:20:28.041617 29881 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:28.029644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57396 (local address 127.6.209.253:45055)
0926 15:20:28.029760 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:28.029764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:28.040290 (+ 10526us) server_negotiation.cc:407] Connection header received
0926 15:20:28.040508 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:28.040513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:28.040593 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:28.040691 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:28.041447 (+ 756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57396: BlockingRecv error: recv got EOF from 127.0.0.1:57396 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 15:20:28.054358 29881 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:28.050259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57406 (local address 127.6.209.253:45055)
0926 15:20:28.050434 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:28.050439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:28.053265 (+ 2826us) server_negotiation.cc:407] Connection header received
0926 15:20:28.053413 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:28.053417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:28.053479 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:28.053564 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:28.054205 (+ 641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57406: BlockingRecv error: recv got EOF from 127.0.0.1:57406 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260926 15:20:28.066272 29881 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:28.064667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57422 (local address 127.6.209.253:45055)
0926 15:20:28.064779 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:28.064783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:28.065229 (+ 446us) server_negotiation.cc:407] Connection header received
0926 15:20:28.065357 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:28.065362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:28.065428 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:28.065524 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:28.066106 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57422: BlockingRecv error: recv got EOF from 127.0.0.1:57422 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 15:20:28.090762 29881 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:28.088851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57438 (local address 127.6.209.253:45055)
0926 15:20:28.089003 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:28.089008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:28.089453 (+ 445us) server_negotiation.cc:407] Connection header received
0926 15:20:28.089671 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:28.089677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:28.089762 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:28.089870 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:28.090562 (+ 692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57438: BlockingRecv error: recv got EOF from 127.0.0.1:57438 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 15:20:28.138417 29881 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:28.136299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57448 (local address 127.6.209.253:45055)
0926 15:20:28.136459 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:28.136465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:28.136988 (+ 523us) server_negotiation.cc:407] Connection header received
0926 15:20:28.137173 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:28.137179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:28.137281 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:28.137400 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:28.138197 (+ 797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57448: BlockingRecv error: recv got EOF from 127.0.0.1:57448 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 15:20:28.208098 29881 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:28.206174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57464 (local address 127.6.209.253:45055)
0926 15:20:28.206370 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:28.206376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:28.206809 (+ 433us) server_negotiation.cc:407] Connection header received
0926 15:20:28.206990 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:28.206998 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:28.207094 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:28.207215 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:28.207881 (+ 666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57464: BlockingRecv error: recv got EOF from 127.0.0.1:57464 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 15:20:28.341750 29881 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:28.339885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57466 (local address 127.6.209.253:45055)
0926 15:20:28.340127 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:28.340135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:28.340388 (+ 253us) server_negotiation.cc:407] Connection header received
0926 15:20:28.340632 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:28.340641 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:28.340709 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:28.340836 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:28.341529 (+ 693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57466: BlockingRecv error: recv got EOF from 127.0.0.1:57466 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 15:20:28.520715 29972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "366534b8b9074bb0b56443a653805985" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "366534b8b9074bb0b56443a653805985" member_type: VOTER last_known_addr { host: "127.6.209.254" port: 45533 } } peers { permanent_uuid: "3be54a518a6845d5bcb4030f2bbcc7c7" member_type: VOTER last_known_addr { host: "127.6.209.253" port: 45055 } } peers { permanent_uuid: "4ceb7c602e5641e9808e233ac671148e" member_type: VOTER last_known_addr { host: "127.6.209.252" port: 44909 } } }
I20260926 15:20:28.521059 29972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 366534b8b9074bb0b56443a653805985 [sys.catalog]: This master's current role is: LEADER
W20260926 15:20:28.602566 29881 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:20:28.600751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57482 (local address 127.6.209.253:45055)
0926 15:20:28.600984 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:28.600992 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:28.601290 (+ 298us) server_negotiation.cc:407] Connection header received
0926 15:20:28.601487 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:28.601494 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:28.601571 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:28.601672 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:20:28.602334 (+ 662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57482: BlockingRecv error: recv got EOF from 127.0.0.1:57482 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 15:20:28.641175 29970 catalog_manager.cc:1259] Loaded cluster ID: 9f024d388b8945f888039601df7905a4
I20260926 15:20:28.641331 29970 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4ceb7c602e5641e9808e233ac671148e: loading cluster ID for follower catalog manager: success
I20260926 15:20:28.644773 29970 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4ceb7c602e5641e9808e233ac671148e: acquiring CA information for follower catalog manager: success
I20260926 15:20:28.646694 29970 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4ceb7c602e5641e9808e233ac671148e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 15:20:28.992303 30022 catalog_manager.cc:1259] Loaded cluster ID: 9f024d388b8945f888039601df7905a4
I20260926 15:20:28.992476 30022 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3be54a518a6845d5bcb4030f2bbcc7c7: loading cluster ID for follower catalog manager: success
I20260926 15:20:28.994726 30022 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3be54a518a6845d5bcb4030f2bbcc7c7: acquiring CA information for follower catalog manager: success
I20260926 15:20:28.996330 30022 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3be54a518a6845d5bcb4030f2bbcc7c7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 15:20:29.126284 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:29.119751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57484 (local address 127.6.209.253:45055)
0926 15:20:29.120515 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:29.120521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:29.120546 (+ 25us) server_negotiation.cc:407] Connection header received
0926 15:20:29.120643 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:29.120649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:29.120873 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:29.121015 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:20:29.121392 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:29.122474 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:29.123632 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:29.124250 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:29.124363 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:29.124932 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:20:29.125609 (+ 677us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:20:29.125743 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:29.125913 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 15:20:29.125988 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":513,"threads_started":1}
I20260926 15:20:29.136423 30092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:29.131267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50606 (local address 127.6.209.252:44909)
0926 15:20:29.132052 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:29.132056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:29.132068 (+ 12us) server_negotiation.cc:407] Connection header received
0926 15:20:29.132145 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:29.132149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:29.132274 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:29.132424 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:20:29.132798 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:29.133466 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:29.134611 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:29.134960 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:29.135075 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:29.135477 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:20:29.135900 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:20:29.136055 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:29.136109 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 15:20:29.136176 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":613,"threads_started":1}
I20260926 15:20:29.149592 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:29.141646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45332 (local address 127.6.209.254:45533)
0926 15:20:29.142683 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:29.142688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:29.142705 (+ 17us) server_negotiation.cc:407] Connection header received
0926 15:20:29.142796 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:29.142801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:29.142966 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:29.143517 (+ 551us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:20:29.143708 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:29.144579 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:29.146444 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:29.147062 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:29.148251 (+ 1189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:29.148402 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:20:29.148727 (+ 325us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:20:29.148844 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:29.149255 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 15:20:29.149331 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":931,"thread_start_us":809,"threads_started":1}
I20260926 15:20:29.153615 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 29736
I20260926 15:20:29.153863 30092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:29.143209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50614 (local address 127.6.209.252:44909)
0926 15:20:29.147657 (+ 4448us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:29.147666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:29.148211 (+ 545us) server_negotiation.cc:407] Connection header received
0926 15:20:29.148364 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:29.148369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:29.148443 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:29.148525 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:20:29.148946 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:29.150059 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:29.152088 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:29.152627 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:29.152711 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:29.153228 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:20:29.153323 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:20:29.153375 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:29.153559 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 15:20:29.153624 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4290}
I20260926 15:20:29.154402 29798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:20:29.155251 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:20:29.141562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57486 (local address 127.6.209.253:45055)
0926 15:20:29.141727 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 15:20:29.141732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:20:29.147638 (+ 5906us) server_negotiation.cc:407] Connection header received
0926 15:20:29.149112 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:20:29.149117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:20:29.149191 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:20:29.149246 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:20:29.149895 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:29.151097 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:29.153021 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:20:29.153579 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:20:29.153649 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:20:29.154595 (+ 946us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:20:29.154688 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:20:29.154747 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 15:20:29.154935 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 15:20:29.155005 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 15:20:29.261441 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 29736
I20260926 15:20:29.288229 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 29810
I20260926 15:20:29.289296 29872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:20:29.394233 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 29810
I20260926 15:20:29.412443 6983 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu pid 29884
I20260926 15:20:29.413493 29946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:20:29.509274 6983 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXn57Q/build/asan/bin/kudu with pid 29884
2026-09-26T15:20:29Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4474 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8894 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (335052 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST